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ABSTRACT 

This  thesis  evaluates  the  Navy's  test  and  evaluation 
(T&E)  process  for  aircraft  weapon  systems,  identifies  prob- 
lems, and  proposes  a  reorganization  of  the  Navy's  T&E 
agencies.   Many  changes  have  been  made  to  Improve  the  T&E 
process,  but  some  of  these  have  resulted  in  costly  ineffi- 
ciencies.  Contributing  problems  include:   the  Navy's 
"independent"  test  agency  being  tasked  more  on  the  basis  of 
its  apparent  independence  than  on  its  testing  expertise, 
lack  of  actual  independence  of  the  many  test  agencies, 
excessive  testing  duplication,  inadequate  testing  expertise, 
lack  of  operational  test  objectives,  inadequate  test  planning, 
lack  of  a  weapon  system  T&E  school,  and  numerous  T&E  manage- 
ment problems. 

The  establishment  of  a  Naval  T&E  Command  which  is  organi- 
zationally separate  from  the  developing  agency  and  the  OPNAV 
program  sponsor  is  proposed.   The  Naval  T&E  Command  would  be 
responsible  for  all  T&E  conducted  by  the  Navy  with  the 
exception  of  that  done  by  Navy  Laboratories.   The  Naval  T&E 
Command  would  have  separate  divisions  responsible  for 
development  T&E  and  operational  T&E,  and  the  developing 
agency  could  task  the  T&E  Command  to  perform  any  type  of  T&E. 
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I.   INTRODUCTION 

A.   GENERAL 

During  the  past  decade  all  phases  of  the  Department  of 
Defense  (DOD)  weapon  system  acquisition  process  have  been 
under  increasing  criticism.   Frequent  cost  growth,  coupled 
with  poor  operational  performance  of  many  weapon  systems 
when  they  entered  operational  service,  has  been  the  main 
impetus  for  the  appointment  of  numerous  committees  and 
commissions  to  study  the  weapon  system  acquisition  process. 
The  President's  Blue  Ribbon  Defense  Panel,  the  Government 
Accounting  Office,  the  Congress's  Commission  on  Government 
Procurement,  and  the  Armed  Services  Committees  of  the  House 
of  Representatives  and  Senate  have  consistently  identified 
test  and  evaluation  as  a  major  problem  area. 

The  current  DOD  major  system  acquisition  policy  pro- 
hibits the  "total  package  procurement"  concept  that  was  in 
vogue  in  the  1960's.   Concurrency  between  development  and 
production  is  being  minimized,  and  a  determination  of  opera- 
tional suitability,  including  logistic  support  requirements, 
is  required -before  a  production  decision  is  made.   This 
change  in  system  acquisition  philosophy  has  resulted  in 
greater  emphasis  in  test  and  evaluation.   Consequently, 
problems  existing  in  DOD's  test  and  evaluation  (T&E)  process 
will  have  an  even  greater  effect  on  the  acquisition  of  major 
weapon  systems  than  ever  before. 
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It  has  been  less  than  three  years  since  the  major  change 
to  the  DOD  weapon  system  acquisition  process  was  implemented 
by  DOD  Directive  5000.1,  entitled  "Acquisition  of  Major 
Defense  Systems"  [Ref.  1].   Unfortunately,  reforming  the 
weapon  system  acquisition  process  is  an  undertaking  which 
does  not  lend  itself  to  instantaneous  change.   By  now,  most 
of  the  implementing  DOD  Directives  and  Navy  instructions 
have  been  written,  but  changes  in  the  area  of  test  and 
evaluation  are  still  being  made  and  others  comtemplated  in 
order  to  more  effectively  implement  the  acquisition  philos- 
ophy that  is  embodied  in  DOD  Directive  5000.1.   Many  problems 
still  exist  in  the  Navy's  T&E  process, and  new  problems  are 
being  created  by  the  implementation  of  the  new  DOD  acquisi- 
tion philosophy. 

As  a  result  of  the  criticism  that  has  been  directed  at 
T&E,  many  people  believe  that  improved  and  expanded  test  and 
evaluation  will  be  the  most  significant  step  towards  elimi- 
nating the  cost  growths  and  performance  underruns  that  have 
been  so  typical  of  weapon  system  development  programs. 
Problems  do  exist  in  the  military's  T&E  process  that 
adversely  affect  the  weapon  system  acquisition  process. 
However,  many  of  the  problems  that  exist  in  T&E  are  sympto- 
matic of  problems  existing  throughout  the  weapon  system 
acquisition  environment,  i.e.  Congress,  DOD,  and  industry. 
In  order  to  improve  the  weapon  system  acquisition  process 
attention  must  be  devoted  to  the  entire  acquisition  process 
and  not  merely  to  making  patchwork  corrections  to  each 
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segment  of  the  process  [Ref.  2].   Although  changes  are 
required  to  the  T&E  process,  one  should  not  erroneously 
believe  that  these  changes  alone  will  have  a  significant  effect 
in  solving  the  many  problems  facing  the  weapon  system 
acquisition  process  today. 

B.  PURPOSE 

The  purpose  of  this  thesis  is  to: 

(1)  evaluate  the  Navy's  test  and  evaluation  process  for 
aircraft  weapon  systems, 

(2)  identify  problems  that  exist  in  the  Navy's  current 
implementation  of  test  and  evaluation  of  aircraft 
weapon  systems, 

(3)  propose  a  reorganization  of  the  numerous  agencies  and 
activities  that  conduct  test  and  evaluation  of  Navy 
aircraft  weapon  systems. 

C.  SCOPE 

In  the  broadest  sense  test  and  evaluation  (T&E)  takes 
place  throughout  a  system's  life  cycle  (Pig.  1).   In  the 
Conceptual  Phase  test  and  evaluation  takes  the  form  of  test 
planning  and  to  a  limited  extent,  simulation,  breadboard,  and 
laboratory  testing.   It  isn't  until  the  latter  portions  of 
the  Design  Phase  that  test  and  evaluation  takes  the  more 
commonly  thought  of  form  of  the  testing  of  prototypes, 
models,  and  subsystems.   It  is  during  the  Design  Phase  and 
the  Production  and  Installation  Phase  that  test  and  evalua- 
tion activity  reaches  a  peak,  and  it  is  this  T&E  that  will 
be  the  area  of  concern  of  this  thesis.   T&E  does  not 
terminate  after  the  Production  and  Installation  Phase,  but 
continues  until  the  system  is  retired.   The  T&E  that  occurs 
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during  the  last  two  phases  concentrates  on  determining 
optimal  tactics  to  employ  the  system  and  testing  and  evalu- 
ating changes  that  are  made  to  the  system.   These  phases 
have  minimal  direct  impact  on  the  acquisition  process  and 
will  not  be  an  area  of  major  consideration  in  this  thesis. 

Although  many  problems  exist  with  the  test  and  evaluation 
of  Navy  aircraft,  ships,  and  weapons,  this  thesis  deals 
solely  with  the  Navy's  test  and  evaluation  of  aircraft 
weapon  systems.   Both  the  Navy  and  the  contractor  conduct 
T&E  on  each  aircraft  weapon  system.   This  thesis  will  address 
primarily  the  T&E  conducted  by  the  Navy. 

D.   METHODS  OF  RESEARCH  AND  ANALYSIS 

A  thorough  study  of  the  available  literature  in  the 
field  of  the  management  and  organization  of  test  and  evalu- 
ation was  conducted.   This  was  followed  by  telephone  inter- 
views with  key  personnel  directly  involved  with  Naval 
aircraft  weapon  system  test  and  evaluation. 

The  Annual  Symposium  of  the  Society  of  Experimental  Test 
Pilots  was  attended.   An  entire  major  session  was  devoted  to 
operational  test  and  evaluation.   This  symposium  afforded 
the  opportunity  to  exchange  ideas  in  the  subject  area  with 
numerous  key  personnel  involved  in  Navy,  Air  Force,  Army, 
NASA,  and  contractor  aircraft  systems  test  and  evaluation. 


Aircraft  weapon  systems  is  used  herein  to  refer  to  the 
total  aircraft  including  the  weapon  delivery  system  and  the 
weapon  itself. 
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After  completing  this  preliminary  research  effort,  it 
became  evident  that  a  high  degree  of  parochialism  exists 
throughout  the  Navy's  T&E  community,  with  most  people 
holding  views  that  greatly  favored  the  organization  to  which 
they  were  presently  attached.   Since  the  subject  of  defi- 
ciencies in  the  Navy's  T&E  organizations  is  a  popular  one, 
most  people  in  each  T&E  activity  shared  similar  ideas  in  the 
subject  area.   This  is  perhaps  a  result  of  the  tremendous 
amount  of  dialogue  and  the  large  interchange  of  ideas  that 
has  taken  place  in  this  subject  area. 

Research  was,  therefore,  confined  to  personal  or  tele- 
phone interviews  with  a  limited  number  of  personnel  involved 
in  each  of  the  various  phases  of  Navy  T&E.   Because  of  the 
parochialism  that  exists  within  each  T&E  organization,  it 
was  possible  to  obtain  an  accurate  cross  section  of  that 
organization's  opinions  of  deficiencies  in  Navy  T&E  by 
interviewing  only  a  few  of  its  people. 

Visits  were  made  to  the  Offices  of  the  Secretary  of 
Defense,  the  Offices  of  the  CNO,  the  Naval  Air  Systems 
Command  Headquarters,  the  Naval  Air  Test  Center,  and  the 
Headquarters,  of  the  Operational  Test  and  Evaluation  Force. 
Appendix  A  contains  a  tabulation  of  the  specific  organiza- 
tions with  which  interviews  were  conducted. 

Fifty-seven  people  were  interviewed  either  In  person  or 
by  telephone.   Interviews  varied  in  length  from  fifteen 
minutes  to  three  and  one-half  hours,  and  telephone  conversa- 
tions varied  in  length  from  ten  minutes  to  one  and  one-half 
hours. 
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II.   BACKGROUND 

A.   WHAT  IS  TEST  AND  EVALUATION? 

In  order  to  ensure  a  common  base  for  understanding  the 
contents  of  this  thesis,  it  is  necessary  to  establish  some 
fundamental  definitions.   A  simple,  yet  all-inclusive, 
definition  of  test  and  evaluation  which  is  utilized  herein 
is  the  operation  of  a  system  or  one  or  more  of  its  components 
In  a  manner  as  similar  as  possible  to  its  intended  use  and 
in  an  environment  which  approximates  as  closely  as  possible 
its  operating  environment  for  the  purpose  of  gaining  informa- 
tion to  evaluate  the  performance  of  that  system  or  component. 
The  information  obtained  may  be  used  to  give  a  better  under- 
standing of  the  system's  capabilities,  to  identify  short- 
comings, to  develop  improvements,  to  assist  in  designing 
the  system's  replacements  or  complements,  to  learn  how  to 
support  and  maintain  the  system,  or  for  developing  require- 
ments for  training  personnel  in  the  operation  of  the  system. 

Aircraft  weapon  system  T&E  can  be  divided  into  basically 
two  types  of  testing:   contractor  T&E  and  military  T&E. 
Contractor  T&E  is  conducted  by  the  contractor  as  a  necessary 
step  in  his  design  and  development  of  a  weapon  system  and 
also  as  part  of  the  contract  requirements  specified  by  DOD. 
Military  T&E  is  that  T&E  conducted  by  military  personnel  to 
verify  that  the  system  meets  the  contract  requirements  and 
to  ensure  that  the  system  being  acquired  is  operationally 
suitable . 
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Aircraft  weapon  system  T&E  can  also  be  divided  into  two 
different  categories:   Development  Test  and  Evaluation  (DT&E) 
and  Operational  Test  and  Evaluation  (OT&E) .   Comprehensive 
definitions  of  DT&E  and  OT&E  which  are  in  concurrence  with 
the  most  recent  DOD  Directives  are  given  in  Appendix  B. 
Briefly,  DT&E  is  that  T&E  conducted  by  both  the  contractor 
and  the  military  under  the  auspices  of  the  developing  agency 
for  the  purpose  of  demonstrating  that  the  engineering  design 
and  development  are  complete,  demonstrating  that  the  system 
will  meet  specifications,  determining  whether  the  design 
risks  have  been  minimized  with  respect  to  system  producibility, 
and  estimating  the  system's  military  worth. 

OT&E  is  that  T&E  conducted  by  the  military  to  estimate  a 
system's  prospective  military  worth,  operational  effective- 
ness,  operational  suitability  (including  compatibility, 
interoperability,  reliability,  maintainability,  and  logistic 
and  training  requirements),  and  the  need  for  any  design  modi- 
fications.  It  is  also  used  to  determine  optimum  weapon 
system  employment  tactics  and  force  structure.   OT&E  should 
be  conducted  by  operational  personnel  with  qualifications 
similar  to  those  expected  to  ultimately  operate  the  system, 
and  the  tests  should  be  conducted  in  as  realistic  an  opera- 
tional environment  as  possible. 

In  summary,  DT&E  can  be  divided  into  two  types: 
contractor  conducted  DT&E  and  military  conducted  DT&E. 


Operational  Effectiveness  is  a  measure  of  how  well  a 
system  meets  its  defined  operational  requirements  In  its 
intended  operating  environment. 
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OT&E  can  also  be  divided  into  two  types:   OT&E  conducted  to 
estimate  a  system's  operational  potential  and  OT&E  conducted 
to  determine  how  to  best  utilize  the  system.   As  a  result 
of  having  received  the  most  criticism,  OT&E  is  currently  the 
type  of  T&E  that  is  being  given  major  attention.   The  under- 
lying thesis  is  that  the  validity  of  quantitative  test  data 
improves  as  realism  (testing  in  an  operational/combat  environ- 
ment) is  added  to  the  environment  of  the  test  program.   The 
achievement  of  this  objective  depends  upon  one's  interpreta- 
tion of  realism.   It  is  easy  to  go  overboard  in  the  quest  for 
realism  to  the  extent  of  actually  reducing  the  efficiency  of 
the  test  program  and  adding  needlessly  to  the  cost  of  the 
test  program.   The  current  emphasis  on  OT&E  appears  motivated 
by  an  overzealous  quest  for-  the  achievement  of  realism  in 
the  test  environment. 

B.   TEST  AND  EVALUATION  CRITICIZED 

One  of  the  earliest  criticisms  of  T&E  in  DOD  was  made  by 
the  Military  Aircraft  Panel  of  the  President's  Scientific 
Advisory  Committee  (PSAC)  in  August  of  1969.   This  panel 
recommended  to  the  Secretary  of  Defense  that  an  Operational 
Test  and  Evaluation  (OT&E)  organization  be  established  in 
the  office  of  the  Secretary  of  Defense.   It  further  went  on 
to  say  that  our  lack  of  preparedness  in  previous  wars  could 
have  been  exposed  if  an  adequate  program  for  test  and  evalu- 
ation had  existed  [Ref.  4]. 

During  this  same  time  frame  the  Defense  Science  Board, 
the  Joint  Chiefs  of  Staff,  the  Bureau  of  the  Budget,  the 
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Comptroller  General  of  the  United  States,  and  various 
Congressional  Committees  recognized  that  OT&E  had  been  much 
less  effective  than  necessary.   Since  OT&E  is  the  fianl  step 
in  the  research,  development,  test  and  evaluation  (RDT&E) 
process,  there  appeared  to  be  a  potentially  large  pay-off 
for  performing  OT&E  more  effectively. 

Perhaps,  the  most  noteworthy  of  the  T&E  studies  was 
conducted  by  the  President's  Blue  Ribbon  Defense  Panel 
[Ref.  4].   Citing  the  aforementioned  criticisms  of  T&E  and 
specifically  OT&E,  one  Task  Group  of  the  Blue  Ribbon  Defense 
Panel  performed  an  assessment  of  how  OT&E  was  being  conducted. 
It  Is  interesting  to  note  that  this  study,  which  became  a 
major  driving  force  behind  changes  to  the  overall  DOD  T&E 
process,  looked  only  at  OT&E  rather  than  the  entire  DOD 
T&E  effort. 

The  main  conclusions  were:  XU 

(1)  OT&E  can  and  should  be  planned  and  executed  much  more 
effectively  than  it  has  been  in  recent  years. 

(2)  OT&E  results  have  not  been  made  available  to  or  used  fa 
by  DOD  agencies  which  need  them. 

(3)  There  is  a  requirement  for  an  OT&E  organization  at 
higher-than-service  level. 

(4)  OT&E  within  the  services  is  done  most  effectively  when 
OT&E  Organizations  report  directly  to  the  Chief  of 
Service,  representing  both  the  developer  and  user, 
but  organizationally  independent  of  both. 

(5)  Inadequate  funding  has  adversely  affected  the  conduct 
of  needed  OT&E. 


The  panel  went  on  to  recommend: 


(1)  An  operational  T&E  Group  should  be  established  with 
civilian  leadership  reporting  directly  to  the  Deputy 
Secretary  of  Defense. 
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(2)  A  Defense  Test  Agency  should  be  established  with  broad 
authority  and  responsibility  for  DOD  test  activities 
giving  particular  emphasis  to  OT&E. 

(3)  The  Secretary  of  Defense  should  communicate  to  the 
Military  Departments  the  importance  he  assigns  to  OT&E 
and  his  conviction  that  OT&E  is  best  served  when 
independent  OT&E  organizations  report  directly  to  the 
Service  Chiefs,  Service  Secretaries,  or  both. 

The  Navy  fared  best  of  all  services  in  this  study,  since 
it  had  in  existence  an  independent  OT&E  agency,  its  Opera- 
tional Test  and  Evaluation  Force  (OPTEVPOR) .   Even  though 
the  Blue  Ribbon  Defense  Panel  demonstrated  by  many  of  its 
statements  a  misunderstanding  of  how  the  Navy  conducted  its 
T&E  (see  page  46),  their  recommendations  received  almost 
universal  acceptance. 

In  1971  the  Government  Accounting  Office  (GAO)  responded 
to  Congressional  requests  and  conducted  a  study  of  T&E  for 
major  weapon  systems  [Ref.  5!K   Its  main  conclusions  were: 

(1)  The  practices  used  in  establishing  test  objectives 
were  generally  inadequate. 

(2)  Most  weapon  systems  reviewed  did  not  have  adequate 
plans  for  testing. 

(3)  T&E  for  most  weapon  systems  was  not  accomplished  in 
a  timely  manner. 

(4)  Complete  and  valid  T&E  data  was  not  available  to 
decision-makers  prior  to  key  decision  points  in  the 
acquisition  cycle. 

The  GAO  report  recommended: 

(1)  Completion  of  appropriate  testing  and  evaluation  prior 
to  key  decision  points  in  the  acquisition  cycle. 

(2)  Stringent  controls  over  the  granting  of  any  waivers 
from  required  T&E. 

(3)  Preparation  of  succinct  summary  reports  by  the  testing 
agency  for  all  levels  of  management.   Interested 
management  levels  may  v.'ish  to  comment  on  these  summary 
reports,  however,  they  should  not  be  permitted  to 
change  the  basic  content. 


21 


The  Commission  on  Government  Procurement  in  1972 
echoed  many  of  the  previously  cited  criticisms  [Ref.  2]  of 
DOD  T&E.   Its  emphasis  was  on  the  need  to  establish  an 
operational  test  agency  that  is  independent  of  the  develop- 
ment and  user  organizations. 

The  main  criticisms  coming  from  these  studies  appear  to 
have  been  the  lack  of  adequate  "operational"  test  and 
evaluation  and  the  lack  of  testing  "independent"  of  the 
weapon  system  developers.   Consequently,  the  words  "operational" 
and  "independent"  are  now  used  in  every  directive  or  instruc- 
tion that  addresses  T&E. 

C.   NEW  HIGH  LEVEL  T&E  POLICY 

DOD  Directive  5000.1  of  July  13,  1971  established  a  new 

policy  for  the  acquisition  of  major  defense  systems.  In  an 

attempt  to  correct  deficiencies  in  DOD  T&E  cited  in  the 

previously  discussed  studies,  DOD  Directive  5000.1  states 

the  following  regarding  T&E: 

"Test  and  evaluation  shall  commence  as  early  as 
possible.   A  determination  of  operational  suitability, 
including  logistic  support  requirements,  will  be  made 
prior  to  large-scale  production  commitments,  making 
use  of  the  most  realistic  test  environment  possible 
and  the -best  representation  of  the  future  operational 
system  available.   The  results  of  this  operational 
testing  will  be  evaluated  and  presented  to  the  DSARC 
at  the  time  of  the  production  decision"  [Ref.  1]. 

Memorandum  issued  by  Deputy  Secretary  of  Defense  David 
Packard  provided  interim  guidance  to  the  services  for  imple- 
menting changes  to  their  respective  T&E  systems  until  the 
issuance  of  a  specific  directive  addressing  T&E  (Directive 
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5000.3  was  issued  in  January  1972).   This  comprehensive 
directive  [Ref.  6]  covers  all  phases  of  DOD  test  and  evalu- 
ation and  is  summarized  as  follows: 

(1)  Development  test  and  evaluation  is  to  be  planned, 
conducted,  and  monitored  by  the  developing  agency 
of  the  DOD  Components. 

(2)  In  each  DOD  Component  there  will  be  one  major  field 
agency  separate  and  distinct  from  the  developing/ 
procuring  command  which  will  be  responsible  for 
operational  test  and  evaluation  and  which  will 
report  results  of  its  independent  T&E  directly  to 
the  Military  Service  Chief  or  Defense  Agency 
Director. 

(3)  Operational  testing  should  be  separate  from  develop- 
ment testing;  however,  combined  testing  may  be 
conducted  where  separation  would  cause  delay 
Involving  unacceptable  risk  or  unacceptable  increased 
cost. 

(4)  At  least  an  initial  phase  of  operational  test  and 
evaluation  (IOT&E)  will  be  accomplished  prior  to  the 
first  major  production  decision  sufficient  to  provide 
a  valid  estimate  of  expected  system  operational 
effectiveness  and  suitability. 


DOD  Component  is  a  term  referring  to  each  of  the  main 
organizations  that  comprise  the  Department  of  Defense:   Army, 
Navy,  Air  Force,  and  the  Defense  Supply  Agency,  for  example. 
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(5)  The  Development  Concept  Paper  (DCP)  prepared  for  use 
at  the  time  of  the  Program  Initiation  Decision  for  a 
major  Defense  System  will  identify  the  critical 
questions  and  areas  of  risk  to  be  resolved  by  test 
and  evaluation. 

(6)  When  the  DOD  component  proposes  to  initiate  Full- 
Scale  Development,  it  must  present  to  the  Defense 
Systems  Acquisition  Review  Council  (DSARC)  the  results 
of  T&E  accomplished  to  that  date,  an  updated  state- 
ment of  critical  questions  and  areas  of  risk  still 
needing  test  to  be  resolved,  and  a  detailed  statement 
of  test  plans  and  milestones. 

(7)  Prior  to  the  first  major  production  decision  DSARC 
will  assess  the  adequacy  of  test  results  to  support 
a  production  decision  and  the  adequacy  of  plans  and 
schedules  for  any  remaining  testing. 

(8)  The  duties  of  the  Deputy  Director  Defense  Research 
and  Engineering,  Test  and  Evaluation  (DD(T&E))  are 
delineated.   He  has  across-the-board  responsibility 
for  the  Office  of  the  Secretary  of  Defense  (OSD)  in 
test  and  evaluation  matters. 

One  of  the  strongest  motivating  forces  behind  the  changes 
that  have  taken  place  in  DOD's  test  and  evaluation  process 
has  been  Congress.   The  report  of  the  Senate  Armed  Services 
Committee  from  the  1st  session  of  the  Ninety-Second  Congress 
[Ref.  7]  fully  supported  the  findings  of  the  Blue  Ribbon 
Defense  Panel.   The  Committee  reiterated  weaknesses  in  DOD's 
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operational  test  and  evaluation  —  lack  of  testing  independent 
of  the  weapon  system  developer,  service  opposition  to  inde- 
pendent operational  test  and  evaluation,  lack  of  funds  and 
facilities,  and  lack  of  high-level  attention  and  management. 
The  committee  went  on  to  state  that  insufficient  OT&E  in  the 
past  had  resulted  in  the  production  and  deployment  of  some 
weapon  systems  which  were  too  complex  to  be  effective.   The 
committee  reasoned  that  the  military  was  misled  into  thinking 
its  personnel  would  be  able  to  maintain  the  systems  satis- 
factorily, but  the  level  of  technical  ability  required  to  do 
so  often  exceeded  that  possessed  in  sufficient  quantity  by 
the  military.   The  committee  reasoned  that  adequate  OT&E 
would  have  revealed  this  soon  enough  to  make  appropriate 
changes . 

In  the  same  session  of  Congress,  an  amendment  to  the 
1972  procurement  authorization  bill  (Public  Law  92-156)  was 
passed  [Ref.  8].   This  amendment,  Section  506,  requires  that 
additional  cost,  schedule,  and  performance  data  be  reported 
to  Congress  in  support  of  the  annual  budget  request  and 
periodically  throughout  the  year  before  major  system  procure- 
ment awards,  are  made  [Ref.  9].   The  required  reports  to 
Congress,  called  Congressional  Data  Sheets,  must  include 
data  on  OT&E  results  for  each  weapon  system  each  year  until 
procurement  is  completed.   If  OT&E  has  not  been  conducted, 
a  statement  of  the  reasons  therefore  and  the  results  of  such 
other  T&E  as  has  been  conducted  must  be  included  In  the 
report  to  Congress. 
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The  Fiscal  Year  (FY)  73  budget  request  was  accompanied 
by  the  newly  required  supporting  program  data  reports. 
While  similar  supporting  data  were  provided  to  Congress 
previously,  PL  92-156  Section  506  standardized  this  procedure 
and  made  compliance  mandatory.   In  passing  this  amendment, 
the  intent  of  Congress  appeared  to  be  to  provide  a  means 
of  obtaining  a  broader  range  of  data  as  a  basis  to  make 
informed  judgments  on  the  merits  of  the  programs  and  their 
funding  requirements  [Ref.  9].   However,  these  Congressional 
Data  Sheets  seem  to  be  somewhat  redundant  of  the  information 
contained  in  the  Selected  Acquisition  Reports  (SAR's)  which 
are  also  required  by  Congress  with  copies  sent  to  GAO.   The 
SAR's  have  often  been  late  in  submission.   For  this  and 
other  reasons  SAR's  have  proven  to  be  ineffective  documents, 
but  no  current  move  is  underway  to  abolish  the  SAR.   Although 
Congress  has  not  changed  its  methods  of  acting  on  the  Armed 
Service  Authorizations  or  Appropriations  Bills  since  re- 
ceiving the  Congressional  Data  Sheets,  there  is  confidence 
in  the  Congress  and  the  GAO  that  improved  communications 
on  weapon  system  development  is  being  created  between  the 
Congress  and  DOD. 
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III.   NAVY  IMPLEMENTATION  OF  T&E  POLICY 

A.  GENERAL 

As  stated  in  Section  IIB,  the  Navy  was  the  only  service 
which  had  in  existence  at  the  time  of  the  Blue  Ribbon  Panel 
Study  an  independent  operational  test  and  evaluation  agency. 
This  was  the  Operational  Test  and  Evaluation  Force  (OPTEVFOR), 
which  reported  to  the  Chief  of  Naval  Operations  (CNO).   It 
was  a  relatively  small  organization  of  approximately  1400 
officers  and  enlisted  men  and  did  not  maintain  its  own 
test  ranges  or  instrumentation. 

It  was  obvious  that  the  most  expedient  method  to  comply 
with  the  DOD  directives  would  the  expansion  of  the  role  of 
OPTEVFOR  to  do  the  required  OT&E.   Appendix  C  contains  a 
list  of  the  various  instructions  and  directives  that  pertain 
to  Navy  Aircraft  Weapon  System  test  and  evaluation.   The 
basic  content  with  regards  to  T&E  of  a  few  of  these  Navy 
originated  instructions  will  be  discussed  in  order  to  show 
the  Navy's  method  of  implementing  the  broad  T&E  guidelines 
given  by  Sec  Def  in  DOD  Directives  5000.1  and  5000.3. 

B.  SECNAV  INSTRUCTION  5000.1  [Ref  10] 

SECNAV  Instruction  5000.1  is  the  Navy's  implementing 
instruction  of  DOD  Directive  5000.1.   It  assigns  Commander 
Operational  Test  and  Evaluation  Force  (COMOPTEVFOR)  as  the 
Navy's  independent  test  agency  for  the  required  operational 
test  and  evaluation.   The  general  sequence  of  test  and 
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evaluation  events  under  the  new  acquisition  philosophy  is 

given  as  follows: 

"1.   Laboratory/contractor  preliminary  T&E  of  bread- 
board of  demonstration  hardware  during  the  conceptual 
effort . 

2.  Contractor/development  activity  T&E  of  subsystems 
and/or  full-scale  demonstrator  hardware  during  full- 
scale  development. 

3.  Technical  T&E  conducted  by  the  contractor  with 
Navy  participation  during  pre-production/production. 

4.  IOT&E  (Initial  Operational  Test  and  Evaluation) 

by  or  with  the  active  participation  of  Navy  operational 
forces  prior  to  the  major  production  decision. 

5.  Navy  OPEVAL  (Operational  Evaluation)  prior  to 
approval  for  service  use  and  inventory  acceptance 
(except  for  ships). 

6.  Navy  follow-on  T&E. 

7.  Conduct  of  T&E  by  the  Board  of  Inspection  and 

Survey  (BIS)  and  recommendation  to  CNO  for  service 

acceptability"  [Ref.  10]. 

-  ■ 
C.   OPNAV  INSTRUCTION  3960.8  [Ref.  11] 

OPNAV  Instruction  3960.8,  issued  by  CNO,  is  essentially 
the  basic  Implementing  Instruction  of  DOD  Directive  5000.3- 
OPNAV  Instruction  3960.8  establishes  guidance  and  policy 
for  T&E  of  all  Navy  weapon  systems,  subsystems,  components, 
and  support  systems.   Specific  areas  discussed  in  this  instruc- 
tion that  are  germane  to  this  thesis  are  discussed  below. 
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Development  test  and  evaluation  (DT&E)  is  defined  as  T&E 
conducted  under  the  sponsorship  of  the  developing  agency 
which  is  undertaken  to  facilitate  the  evolution  of  a  system. 
The  primary  objective  of  DT&E  is  given  as  the  generation  of 
essential  and  valid  data  upon  which  to  base  design  decisions 
during  the  development  process. 

Operational  test  and  evaluation  is  defined  as  including 
all  T&E  efforts  undertaken  for  the  purpose  of  obtaining 
operational  information  throughout  the  system's  life  cycle. 
This  T&E  is  used  to  support  both  the  acquisition  process 
and  the  optimum  employment  of  the  system.   OT&E  is  broken 
into  two  phases:   IOT&E  (Initial  Operational  T&E)  and.FOT&E 
(Follow-on  Operational  T&E).1  IOT&E  is  defined  as  that  T&E 
accomplished  by  or  under  the  supervision  of  the  Navy's 
independent  testing  agency  (OPTEVFOR)  prior  to  the  first 
major  production  decision.   FOT&E  is  the  continuing  T&E  of 
a  weapon  system  conducted  under  fleet  conditions  by  opera- 
tional personnel  in  order  to  verify  system  performance, 
validate  correction  of  deficiencies  previously  identified, 
and  refine  tactical  employment  doctrines  and  requirements 
for  personnel  and  training. 


Follow-on  Operational  Test  and  Evaluation  (FOT&E)  is 
defined  in  OPNAV  Instruction  3960.8  to  include  all  OT&E 
following  IOT&E.   The  useage  of  the  term  FOT&E  has  been 
rather  limited  and  is  not  used  in  OPNAV  Instruction  5M0.47D 
or  3930.8B,  both  dated  after  OPNAV  Instruction  3960.8. 
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The  basic  objectives  of  OPTEVFOR  participation  in  T&E 
throughout  a  system's  life  cycle  are  given  as: 

(1)  Ensuring  that  early  and  realistic  operational  tests 
and  evaluations  are  planned. 

(2)  Ensuring  that  appropriate  initial  operational  tests 
and  evaluations  are  conducted  prior  to  scheduled 
decision  milestones,  i.e.  program  initiation 
decision,  full-scale  development  decision,  and 
production  decision 

(3)  Providing  an  independent  assessment  of  the  operational 
effectiveness  and  suitability  of  the  system  to  the 
CNO  [Ref.  11]. 

The  role  of  the  Board  of  Inspection  and  Survey  (BIS) 
for  conducting  acceptance  trials  on  new  ships  and  new  model 
aircraft  prior  to  Navy  acceptance  from  the  contractor  is 
briefly  discussed  in  this  instruction.   OPTEVFOR  conducts 
OT&E  to  determine  operational  effectiveness  and  suitability, 
and  BIS  conducts  T&E  to  determine  acceptance  for  service 
use.   It  is  recognized  that  these  are  closely  related 
objectives,  and  this  instruction  directs  that  care  be  taken 
to  preclude  duplicate  testing.   One  method  outlined  to 
accomplish  this  is  the  preparation  of  the  Test  and  Evaluation 
Master  Plan  (TEMP)  which  is  the  responsibility  of  the 
Project  Manager  who  works  for  the  Chief  of  Naval  Material. 

Figure  2,  adopted  from  OPNAV  Instruction  3960.8,  shows 
the  normal  sequence  of  test  events  in  the  acquisition  of 
an  aircraft  weapon  system. 

The  Director,  RDT&E  (OP-098)  is  assigned  as  the  focal 
point  for  T&E  policy  and  guidance  for  CNO.   Under  OP-O98 
the  Test  and  Evaluation  Division,  OP-983,  is  assigned  the 
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specific  ressponslbilities  of  coordinating  the  overall  Navy 
T&E  effort. 

D.  OPNAV  INSTRUCTION  5^40. 47D  [Ref.  12] 

OPNAV  Instruction  5440. 47D  promulgates  the  mission  and 
functions  of  0PTEVF0R.   OPTEVFOR's  mission  is  to  "opera- 
tionally test  and  evaluate  specific  weapon  systems,  ships, 
aircraft  and  equipments,  including  procedures  and  tactics, 
where  required;  and,  when  directed  by  CNO,  assist  development 
agencies  in  the  accomplishment  of  necessary  Development  T&E" 
[Ref.  12]. 

The  Director,  RDT&E,  (OP-098)  is  the  OPNAV  focal  point 
for  the  Commander,  0PTEVF0R.   C0M0PTEVF0R  is,  however, 
authorized  direct  access  to  CNO  and  all  subordinate  offices 
as  necessary  to  perform  his  assigned  mission. 

E.  SYNOPSIS  OF  NAVY  IMPLEMENTATION 

The  Navy  had  the  simplest  job  of  any  of  the  Services  in 
implementing  changes  to  its  T&E  process  so  that  it  would 
conform  to  the  new  acquisition  philosophy.   Basically,  the 
only  organizational  change  made  by  the  Navy  was  the  creation 
of  the  relatively  small  (10  officers)  OP-983,  Test  and 
Evaluation  Division  in  the  office  of  CNO's  Director  of 
RDT&E.   Figure  3  shows  the  present  organizational  structure 
for  the  major  activities  that  are  involved  in  aircraft 
weapon  system  T&E.   Appendix  D  contains  a  summary  of  the 
functions  and  responsibilities  of  each  of  the  major  test 
activities . 
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The  role  of  OPTEVFOR  has  been  greatly  expanded  and  the 
roles  of  the  other  Navy  T&E  agencies  have  remained  essen- 
tially unchanged  by  the  implementation  of  the  .new  T&E 
instructions. 

The  Program  Managers  in  the  S-3A  and  P— 14  programs 
faced  the  challenge  of  modifying  their  test  programs  to 
include  the  new  OT&E  requirements.   In  both  programs  the 
leadership  displayed  by  the  program  office  in  soliciting 
the  necessary  cooperation  from  the  many  T&E  agencies  was 
the  primary  ingredient  which  enabled  the  injection  of  the 
increased  OPTEVFOR  testing  which  had  not  been  planned  for 
previously.   In  the  F-lH   program  the  necessary  cooperation 
was  obtained  through  the  formal  establishment  of  the  Joint 
Evaluation  Team  (JET)  which  was  composed  of  test  personnel 
from  the  Naval  Air  Test  Center,  Naval  Missile  Center,  and 
Air  Test  and  Evaluation  Squadron  Four  (VX-^J  who  works  for 
OPTEVFOR)  [Ref.  13]. 
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IV.   ASPECTS  OF  CURRENT  NAVY  AIRCRAFT  T&E  POLICY 

Many  deficiencies  still  exist  in  the  Navy's  T&E  process, 
and  others  are  being  generated  as  a  result  of  the  new  T&E 
policies.   This  section  will  discuss  some  of  the  deficiencies 
that  exist  In  the  Navy's  T&E  process  as  well  as  some  of  its 
enhancing  characteristics  which  have  not  been  changed. 

A.   ORGANIZATIONAL  DEFICIENCIES 

Because  of  the  fragmented  manner  in  which  aircraft  T&E 
is  organized  in  the  Navy,  many  problems  result.   These 
include: 

(1)  The  user  is  inadequately  represented 

(2)  Lack  of  independence  of  the  T&E  organizations 

(3)  Duplication  of  T&E  responsibilities 

(lJ)  Inadequate  central  management  of  NAVAIR  T&E  activities 

(5)  Adequate  testing  is  not  assured  for  non-major 
procurements 

(6)  Requirement  for  independent  T&E  often  ignores  test 
agency  expertise 

(7)  No  standardization  in  method  of  funding  T&E  agencies 

(8)  Navy  test  range  management  is  too  fragmented. 

These  deficiencies  are  discussed  in  the  following  sections. 

1 .   The  'User  is  Inadequately  Represented 

In  each  Navy  procurement  a  dialogue  that  is  vital 
for  success  is  that  between  the  user  and  the  producer.   Two 
such  dialogues  must  exist:   an  external  dialogue  in  which 
the  Navy  is  the  user,  and  the  contractor  is  the  producer, 
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and  an  Internal  dialogue  In  which  the  operating  command  is 
the  user  and  the  acquisition  command  is  the  producer. 

In  the  internal  user-producer  relationship  the  user 
Is  responsible  for  the  operation  and  support  of  a  system 
and  is  consequently  also  responsible  for  developing  and 
stating  the  needs  and  concepts  for  the  system.   In  other 
words  the  user  provides  the  input  requirements  to  which  the 
producer  designs  the  system.   The  producer  is  then  responsi- 
ble for  the  design,  development,  production,  and  installation 
of  a  system  that  meets  the  requirements  of  the  user  and 
that  can  be  operated  and  supported  cost-effectively.   A 
constant  dialogue  must  be  maintained  between  the  internal 
and  external  user  and  producer  throughout  the  system's 
life  cycle  [Ref.  3]. 

The  external  user-producer  relationship  is  well 
defined  and  does  not  appear  to  present  any  significant  prob- 
lems in  regards  to  responsibility.   However,  the  internal 
user-producer  relationship  within  the  Navy  is  not  so  well 
defined.   The  Chief  of  Naval  Material  (CNM)  represents  the 
producer.   The  management  of  the  development  and  production 
of  a  weapon  system  becomes  the  responsibility  of  one  of 
CNM's  System  Commands  and  the  specific  responsibility  of 
a  Program  Manager.   The  Office  of  the  CNO  (OPNAV)  represents 
the  user.   Within  OPNAV  the  Director,  RDT&E  (OP-O98)  initially 
becomes  the  Program  Sponsor  for  the  development  of  the 
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system,  but  he  will  eventually  lose  this  sponsorhsip  to  the 
ultimate  user  sponsor.   On  aircraft  programs  it  is  the  DCNO 
for  AIR  (OP-05)  who  will  become  the  ultimate  user  sponsor, 
and  he  almost  always  assigns  a  Program  Coordinator  from 
his  office  to  coordinate  each  program.   The  OPNAV  Program 
Sponsor  is  responsible  for  determining  program  objectives, 
time  phasing  and  support  requirements,  and  for  appraising 
progress,  readiness,  and  military  worth  of  the  weapon  sys- 
tem.  The  OPNAV  role  as  the  representative  of  the  user 
(i.e.  the  operating  forces)  appears  well  established  up  to 
the  time  of  a  favorable  DSARC  program  initiation  decision. 
Thereafter,  OPNAV  often  becomes  more  of  a  proponent  of  the 
system  being  developed  rather  than  an  impartial  representa- 
tive of  the  operating  forces.   This  is  especially  true  in 
today's  environment  in  which  Congress  is  taking  a  more 
active  role  in  each  major  weapon  system's  development,  and 
the  CNO  is  compelled  to  frequently  justify  each  Navy  program 
to  Congress.   The  opportunistic  exploration  that  is  often 
made  by  some  members  of  Congress  over  any  hint  of  a  serious 
problem  in  a  weapon  system's  development  adds  further  to 
OPNAV s  serving  only  as  a  proponent  of  the  system  and  to 
conceal  any  of  its  problems. 


The  transition  of  Program  Sponsorship  from  OP-O98  to 
OP-05  should  normally  take  effect  upon  the  decision  to 
procure  the  weapon  system  for  service  use.   Of  necessity 
there  is  a  large  period  of  dual  interest  and  activity, 
but  only  one  Program  Sponsor  has  final  responsibility. 
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OPTEVFOR,  working  for  OP-098,  Is  a  test  agency 
organizationally  independent  from  the  developing  agency, 
CNM.   Fortunately,  OPTEVFOR  is  not  put  in  a  position  like 
that  of  the  Program  Sponsor  where  it  must  conceal  system 
deficiencies.   OPTEVFOR  is  able  to  test  and  evaluate  weapon 
systems  while  truly  representing  the  user  in  determining 
if  the  weapon  system  is  operationally  suitable  and  whether 
it  fulfills  an  operational  need. 

OPTEVFOR,  headed  by  a  Rear  Admiral,  reports  to 
CNO's  Director  of  RDT&E,  a  Vice  Admiral,  who  is  often  forced 
into  a  role  where  he  must  conceal  weapon  system  development 
problems  or  risk  losing  the  program  at  the  Congressional 
budget  hearings.   Therefore,  much  of  the  valuable  user 
dialogue  that  is  generated  by  OPTEVFOR  during  the  latter 
stages  of  a  weapon  system's  development  is  lost  through  the 
OP-098  filter. 

The  internal  user-producer  dialogue  during  the 
latter  stages  of  a  weapon  system's  development  often  becomes 
too  producer  oriented.   The  OPNAV  program  sponsor,  forced 
into  a  system  proponent  role,  becomes  an  ineffective 
representative  of  the  user  operational  forces. 

The  Navy's  unilineal  system,  in  which  the  operating 
forces  and  the  Navy's  acquisition  forces  report  to  the 
CNO,  makes  a  clean  separation  between  the  internal  user 
and  producer  forces  extremely  difficult  to  achieve.   The 
creation  of  a  bilineal  system  in  which  the  Chief  of  Naval 
Material  (heading  the  producer  forces)  and  the  Chief  of  Naval 
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Operations  (heading  the  operating  forces)  report  to  a  Navy 
Chief  of  Staff  appears  to  be  the  only  complete  solution  to  ) 
this  user-producer  dilemma.   However,  much  improvement  can 
be  made  under  the  current  unilineal  system.   It  is  beyond 
the  scope  of  this  thesis  to  discuss  a  change  as  drastic  as 
would  be  the  creation  of  a  bilineal  system. 

2.   Lack  of  Independence  of  the  T&E  Organizations 

The  need  for  independent  T&E  has  become  a  well  agreed 
upon  fact.   As  discussed  in  Section  IIB  much  of  the  T&E 
conducted  by  the  Navy  has  been  criticized  as  being  non- 
independent  in  that  the  test  agency  worked  for  the  developing 
agency. 

At  the  working  level  within  the  Navy's  T&E  structure 
there  is  little  concern  for  lack  of  independence.   No  evi- 
dence indicates  that  T&E  conducted  by  any  test  agency  is 
biased  and  that  the  project  team's  reported  results  are  not 
accurate  results  of  tests  conducted  to  the  best  of  their 
ability.   Navy  program  managers  attest  that  the  T&E  conduc- 
ted by  the  NAVAIR  T&E  agencies  is  "independent",  i.e.  the 
P.  M.  is  unable  to  influence  the  results.   No  one  has  really 
questioned  the  independence  of  the  test  personnel  and 
their  ability  to  conduct  and  report  the  actual  test  results 
they  obtain.   However,  Congress  has  viewed  the  NAVAIR  T&E 
agencies  as  not  being  independent,  since  the  management 
wiring  diagram  shows  them  working  for  the  developing  agency. 
The  accurate  assumption  is  that  NAVAIR  has  the  power  to 
completely  disregard  its  DT&E  agency  test  results,  and  the 
DT&E  agency  has  no  recourse. 
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A  complete  organizational  separation  is  required 
to  achieve  the  independence  that  is  being  sought  by  each  of 
the  critics  of  the  Navy's  T&E  process.   This  is  exactly  what 
the  Navy  did  in  assigning  the  increased  OT&E  responsibility 
to  OPTEVFOR,  an  agency  that  is  organizationally  separated 
from  the  aircraft  developing  agencies,  NAVAIRSYSCOM  and 
NAVMAT . 

However,  OPTEVFOR  is  not  a  truly  independent  test 
organization.   The  Commander,  OPTEVFOR  reports  to  OP-O98, 
who  is  the  program  sponsor  and  was  shown  in  the  preceding 
section  to  play  a  producer-oriented  system  proponent  role 
during  OPTEVFOR' s  involvement  in  the  system's  development. 
Although  OPTEVFOR  reports  the  results  of  its  T&E  directly 
to  CNO,  which  is  an  acceptable  procedure  under  DOD  Directive 
5000.3,  the  OPTEVFOR  reporting  Chain-of-Command  (COMOPTEVFOR  - 
OP-O98  -»>  OP-09  ■*  CNO)  is  not  within  the  intent  of  the  current 


DOD  philosophy.   If  COMOPTEVFOR  must  report  to  and  through 
OP-098,  he  cannot  be  considered  legitimately  independent  of 
the  developing  agencies  [Ref.  14],  since  he  cannot  convey  the 
results  of  his  T&E  with  enough  force  to  effect  a  change  in 
the  system's  development. 

An  examination  of  OPTEVFOR 's  funding  and  available 
physical  resources  shows  further  the  inability  of  OPTEVFOR 
to  perform  as  a  truly  independent  test  agency.   OPTEVFOR 
does  not  maintain  complete  fiscal  control  and  management  of 
all  funding  required  for  OT&E.   In  fact,  all  applicable  costs 
expended  for  OT&E  have  never  been  separately  identified. 
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Without  control  of  all  the  OT&E  funding,  It  would  appear 
that  OPTEVFOR  cannot  independently  plan  and  conduct  OT&E. 
In  any  program  whoever  controls  the  funding  controls  the 
destiny  of  the  program. 

Lack  of  control  of  the  major  portion  of  the  physical 
resources  required  for  OT&E  (instrumentation,  ranges,  data 
processing  centers,  simulators,  and  test  aircraft  )  further 
reduces  the  independence  of  OPTEVFOR.   A  major  portion  of 
the  physical  resources  utilized  by  OPTEVFOR  is  controlled 
by  the  developing  agency.   Yet,  no  formal,  mechanism  exists 
for  the  development  of  integrated  plans  for  the  utilization 
of  these  limited  resources  by  OPTEVFOR  and  the  DT&E  agencies. 
3.   Duplication  of  T&E  Responsibilities 

Much  duplication  of  responsibilities  exists  among 
the  major  participants  in  the  Navy's  T&E  process:   the 
contractor,  NAVAIR  T&E  agencies,  OPTEVFOR,  and  Board  of 
Inspection  and  Survey  (BIS).   The  limited  availability  of 
personnel  and  physical  T&E  resources  dictates  that  the  Navy 
must  eliminate  needless  duplication  in  order  to  achieve 
optimal  utilization.   Flight  testing  is  the  most  expensive 
type  of  aircraft  testing, and  this  is  the  area  that  will  be 
given  primary  attention. 

The  contractor  is  required  to  demonstrate  satisfac- 
torily each  segment  of  an  aircraft's  flight  envelope 
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1Test  aircraft  are  not  permanently  assigned  to  OPTEVFOR 
for  IOT&E  but  only  for  follow-on  T&E. 
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(airspeed,  altitude,  normal  accelerations,  gross  weights, 
etc.)  before  Navy  test  pilots  are  permitted  to  operate  the 
aircraft  within  each  flight  regime  [Ref.  15].   Although 
valuable  in  promoting  safety  and  obtaining  additional  data 
under  identical  test  conditions,  this  regulation  causes  many 
Navy  test  flights  to  be  mere  duplication  of  contractor  test 
flights. 

Contractors  are  routinely  required  to  redemonstrate 
compliance  with  the  aircraft's  flight  structural  and  perform- 
ance requirements  at  the  Naval  Air  Test  Center,  Patuxent 
River,  Maryland,  after  they  have  already  performed  the  same 
tests  at  their  own  facility.   This  procedure  is  continually 
claimed  by  contractors  to  be  a  superfluous  expense  which 
could  be  eliminated  by  having  the  Navy  witness  these  demon- 
strations at  the  contractor's  facility.   The  Naval  Air  Test 
Center  has  been  an  avid  supporter  of  the  need  for  this 
duplication  of  contractor  tests  for  these  reasons: 

(1)  More  credence  can  be  placed  upon  results  when  tests 
are  conducted  at  the  Navy's  facility  rather  than  the 
contractor' s. 

(2)  These  limited  demonstrations  verify  the  contractor's 
instrument  calibrations  and  put  more  credibility  on 
all  data  collected  at  the  contractor's  facility. 

Compounding  the  contractor  flight  demonstration 
problem  is  the  inability  of  the  various  Navy  Plant  Repre- 
sentative Offices  (NAVPRO's)  to  witness  competently  all  of 
the  contractors'  flight  test  demonstrations.   This  is  the 
result  of  the  limited  flight  test  and  aerodynamic  expertise 
that  is  available  at  the  NAVPRO's.   The  various  test  agencies 
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would,  therefore,  be  required  to  send  numerous  qualified 
personnel  to  witness  these  types  of  demonstrations  if  they 
were  conducted  only  at  the  contractor's  facility. 

The  distinction  between  OT&E  and  DT&E  implies  two 
markedly  different  approaches  to  T&E  which  requires  the  need 
for  separate  test  programs  [Ref.  16].   Therefore,  OT&E  has 
been  conducted  primarily  by  OPTEVFOR  and  DT&E  by  the  NAVAIR 
T&E  agencies.   Prior  to  DOD  Directive  5000.1,  there  was  a 
certain  amount  of  duplication  in  the  test  programs  of 
OPTEVFOR  and  the  NAVAIR  T&E  agencies.   This  was  unavoidable 
since  completely  different  people  were  conducting  tests  on 
the  same  system,  and  a  learning  curve  had  to  be  developed 
by  each  of  the  respective  testers.   This  same  type  of  dupli- 
cation exists  today  and  will  continue  to  exist  as  long  as 
separate  agencies  are  performing  tests  on  the  same  weapon 
systems . 

Before  DOD  Directive  5000.1,  the  tests  were  conducted 
in  a  sequential  order  and  little  overlap  existed  between  the 
time-phasing  of  the  tests  conducted  by  the  NAVAIR  T&E  agencies 
and  OPTEVFOR.   With  the  current  requirement  of  conducting 
Initial  Operational  Test  and  Evaluation  (IOT&E)  prior  to  the 
production  decision,  the  amount  of  time  overlap  between 
OPTEVFOR  and  NAVAIR  testing  has  greatly  expanded.   Figure  4 
gives  a  presentation  of  the  conduct  of  the  various  phases  of 
T&E  before  DOD  5000.1  and  today. 

The  increased  time  overlap  has  increased  the  amount 
of  duplication  in  testing.   In  the  case  of  an  aircraft  both 
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OPTEVFOR  and  NAVAIR  test  agency  flight  crews  (each  agency 
needs  more  than  one  flight  crew  to  assure  corroboration  of 
subjective  evaluations)  must  conduct  tests  concurrently 
throughout  the  aircraft's  evolution  —prototype  to  production 
model  —  and  the  many  iterations  of  design  in  between. 
Valuable  flight  test  time  must  consequently  be  devoted  to 
developing  each  flight  crew's  experience  with  each  iteration 
of  design  in  the  aircraft  prior  to  conducting  meaningful 
tests. 

The  most  obvious  duplication  in  Navy  T&E  is  the 
very  similar  testing  objectives  that  exist  for  both  BIS  and 
OPTEVFOR.   BIS,  utilizing  NAVAIR  T&E  agencies  to  perform  the 
test  and  evaluation,  is  responsible  to  "discover  and  report 
deficiencies  at  the  earliest  possible  time  so  that  corrective 
action  can  be  taken  to  ensure  that  aircraft  delivered  to 
operating  units  will  be  capable  of  carrying  out  the  intended 
mission..."  and  to  "recommend  to  the  Secretary  of  the  Navy 
the  conditions  of  acceptance  or  rejection  of  a  model  aircraft 
for  service  use"  [Ref.  17].   The  majority  of  the  pilots 
conducting  the  tests  for  BIS  are  graduates  of  the  U.  S.  Navy 
Test  Pilot  School  where  they  are  taught  to  test  for  service 
suitability  first  and  contract  compliance  second. 

OPTEVFOR  is  responsible  through  its  IOT&E  for  asses- 
sing the  operational  suitability  of  an  item  prior  to  the 
production  decision  and  to  "evaluate  the  operational  effec- 
tiveness, suitability,  and  capability  of  tested  weapons 
systems  to  meet  stated  needs  and  performance  criteria" 
[Ref.  12]. 
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The  difference  between  BIS's  service  suitability 
testing  and  OPTEVFOR's  operational  suitability  testing  is 
not  apparent.   The  difference  between  BIS's  testing  for 
specification  compliance  and  OPTEVFOR's  testing  against  the 
stated  needs  and  performance  criteria  is  also  not  apparent. 
With  the  testing  objectives  being  so  similar,  one  can  only 
expect  duplication  in  the  testing.   Confusion  often  exists 
as  to  what  testing  is  OT&E  and  what  is  DT&E,  which  tends  to 
conceal  much  of  the  duplication  in  T&E  responsibility.   This 
confusion  is  illustrated  by  a  quote  from  the  Blue  Ribbon 
Defense  Panel  which  erroneously  thought  BIS  was  conducting 
OT&E.   "Before  Navy  aircraft  and  ships  are  approved  for  use 
by  the  operational  forces,  they  are  subjected  to  another 
kind  of  operational  evaluation  which  is  much  closer  to  true 
OT&E.   This  is  done  by  a  Board  of  Inspection  and  Survey  (BIS), 
which  is  composed  of  experienced  officers  with  appropriate 
Navy  backgrounds"  [Ref.  4,  p.  23]. 

In  attacking  the  problem  of  duplication  of  T&E 
responsibilities,  the  objective  should  not  be  to  eliminate 
all  overlap  or  duplication.   It  should,  however,  be  to  ensure 
that  where  such  overlap  or  duplication  exists,  it  is  visible, 
controlled,  purposeful,  and  contributes  to  improving  the 
overall  T&E  process. 

l\.      Inadequate  Central  Management  of  HAVAIR  T&E  Activities 

The  Naval  Air  Systems  Command  Test  and  Evaluation 
Coordinator  was  formally  established  in  1968.   The  Commander, 
Naval  Air  Test  Center,  Patuxent  River,  Maryland,  was  assigned 
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this  additional  duty,  and  was  given  a  T&E  Coordinator's 
staff  to  assist  in  the  performance  of  these  duties  which 
were  separate  and  distinct  from  the  other  functions  performed 
at  Patuxent  River.   The  creation  of  the  T&E  Coordinator 
stemmed  from  the  recognition  of  the  lack  of  a  single  focal 
point  or  reviewing  authority  for  T&E  in  NAVAIR  and  the 
consequential  lack  of  coordination  among  the  NAVAIR  T&E 
agencies  [Ref.  18]. 

The  NAVAIR  T&E  Coordinator's  Charter  [Ref.  19] 
designates  the  T&E  Coordinator  as  the  focal  point  for  all 
NAVAIRSYSCOM  responsibilities  in  the  coordination  of  T&E 
matters.   It  should  be  observed,  however,  that  the  T&E      i 
Coordinator  is  only  a  coordinator  and  has  no  line  management 
authority  over  any  of  the  NAVAIR  T&E  agencies,  except  NATC 
which  he  commands.   The  T&E  Coordinator  is  responsible  for 
assisting  in  the  preparation  of  test  planning  in  order  to 
ensure  optimum  utilization  of  resources  and  minimum  duplica- 
tion of  tests.   However,  the  T&E  Coordinator  only  recommends 
which  agency  should  do  the  tests,  and  NAVAIR  makes  the  final 
determination. 

The  lack  of  a  single  management  focal  point  for  T&E 
within  the  internal  NAVAIR  organization  continues  to  exist. 
Since  NAVAIR  has  refused  to  delegate  to  the  T&E  Coordinator 
the  authority  to  manage  its  T&E  activities,  many  duplications 
of  testing  responsibilities  exist  and  needless  interagency 
conflict  often  exists.   For  example,  NATC  and  NMC  are 
assigned  similar  missions  [Ref.  15,  P-  G-13]: 
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NATC:   "Coordinate  and  perform  test  and  evaluation 

of  aircraft  weapons  systems,  their  components 
and  related  equipment,  ..." 

NMC:    "To  perform  test,  evaluation,  development  support 
and  exercise  engineering  cognizance  as  assigned 
of  Naval  weapons,  weapon  systems,  and  related 
devices." 

NMC  has  test  responsibility  for  missiles  and  the 
interfaces  with  the  various  launch  platforms  (aircraft) 
while  NATC  has  test  responsibility  for  the  aircraft  and  the 
interfaces  with  the  various  weapons  that  it  will  deliver. 
Because  of  the  large  overlap  in  responsibility,  much  conflict 
has  arisen  in  the  past  between  Weapon  System  Test  (WST  -  a 
division  of  NATC)  and  NMC.   Participants  in  the  F-lVPhoenix 
Missile  Test  program  have  stated  that  more  conflict  existed 
between  these  two  NAVAIR  T&E  agencies  than  between  either 
NATC  or  NMC  and  VX-4  who  reports  to  OPTEVFOR. 

Another  example  of  NAVAIR  duplication  of  testing 

responsibilities  exists  in  the  area  of  aeromedicine.   Below 

is  a  list  of  the  agencies  directly  involved  with  the  T&E 

of  life  support  and  aircrew  protective  equipment: 

Naval  Aerospace  Recovery  Facility,  El  Centro,  CA .  - 

conducts  DT&E  on  escape  and  recovery  systems 
(see  Appendix  D) 

NATC  -  Aeromed  Branch  of  Service  Test  Division 
NMC  -  Life  Support  and  Life  Sciences  Department 

NADC,  Warminster  -  Although  not  reporting  directly  to 

NAVAIR,  NADC  receives  approximately  85  percent 
of  its  funding  from  NAVAIR  and  does  extensive 
development  and  testing  in  the  field  of 
aeromedicine 

Each  of  these  agencies  is  involved  with  the  development  and 

T&E  in  the  aeromedical  area.   The  need  to  have  aeromedical 

capabilities  and  expertise  at  each  of  these  facilities  is 
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questioned  as  needless  duplication  which  is  permitted  to 

exist  and  expand  because  no  central  authority  is  controlling 

NAVAIR's  T&E  activities. 

5.   Requirements  for  Independent  T&E  Often 
Ignore  Test  Agency  Expertise 

LTGEN  Alfred  Starbird  (USA-Ret)  is  the  Deputy 
Director  of  Defense  Research  and  Engineering,  Test  and 
Evaluation  (DD(T&E))  in  the  Office  of  the  Secretary  of 
Defense  (OSD) .   As  DD(T&E)  he  has  become  a  main  driving 
force  in  ensuring  that  each  of  the  Services  conduct  T&E  in\ 
accordance  with  DOD  Directive  5000.3. 

DD(T&E)  is  responsible  for  reporting  to  the  DSARC 
"at  each  major  milestone  decision  point  his  assessment  as  to 
the  adequacy  of  the  identified  critical  issues  and  questions 
to  be  resolved  by  test  and  evaluation,  test  plans  and 
schedules,  and  the  adequacy  of  the  accomplished  T&E  to 
justify  the  action  recommended  for  that  milestone  decision" 
[Ref.  6].   OPTEVFOR,  as  the  Navy's  independent  operational 
T&E  agency,  is  the  main  source  of  DD(T&E)*s  T&E  data. 
Although  much  of  the  development  agency's  T&E  data  are  made 
available  to  DD(T&E),  it  is  the  OPTEVFOR  evaluation  that  is 
required  before  DD(T&E)  will  be  satisfied  that  adequate  T&E 
has  been  accomplished  to  meet  a  milstone  T&E  objective.   In 
the  event  of  a  conflict,  the  OPTEVFOR  opinion  will  take 
precedence . 

As  will  be  discussed  in  Section  IV. B. 4,  OPTEVFOR 
often  lacks  the  technical  expertise  to  conduct  a  particular 
test  of  a  system  or  subsystem.   Yet  DD(T&E)  has,  in  the  past, 
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required  that  OPTEVFOR  conduct  an  evaluation  before  continu- 
ing with  the  development  program,  since  DD(T&E)  was  not 
satisfied  with  only  the  results  from  an  NATC  (a  developer's 
T&E  agency)  evaluation.   In  other  words,  OPTEVFOR  is  being 
tasked  to  conduct  T&E  on  the  basis  of  its  independence 
rather  than  on  its  testing  capability.   The  result  of  such 
actions  is  that  OPTEVFOR  is  being  compelled  to  build  up  its 
technical  expertise  and  expand  its  testing  capabilities  into 
areas  where  this  capability  and  expertise  exists  in  other 
test  agencies. 

The  distinction  between  OT&E  and  DT&E  is  not  always 
apparent.   Many  T&E  milestones  for  aircraft  procurements 
could  easily  be  shown  to  be  met  by  conducting  only  so  called 
development  testing;  e.g.  meeting  a  performance  requirement 
of  maximum  airspeed  or  altitude.   However,  as  long  as  DD(T&E) 
requires  the  opinion  of  the  independent  test  agency,  OPTEVFOR, 
the  Navy  will  be  forced  to  increase  the  capability  of 
OPTEVFOR.   This  will,  in  turn,  further  increase  the  duplica- 
tion of  responsibilities  between  OPTEVFOR  and  the  development 
test  agencies. 

6.   Adequate  Testing  is  Not  Assured  for  Non-Major  y^ 
Procurements 

DOD  Directive  5000.1  and  the  various  implementing 

instructions  pertain  to  major  programs  (RDT&E  cost  in  excess 

of  50  million  dollars  or  production  cost  in  excess  of  200 

million  dollars).   However,  all  the  directives  state  that 

the  principles  are  to  applied  to  all  programs.   Since  DSARC 

is  only  involved  with  major  programs,  there  is  no  exogenous 
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system  to  ensure  that  adequate  and  independent  T&E  is 
conducted  prior  to  making  the  advanced  development  or  the 
production  decision  for  non-major  programs. 

The  lack  of  any  explicit  policy  for  T&E  of  non-major 
systems  has  given  considerable  latitude  to  the  Program 
Manager.   Many  projects  are  not  subject  to  automatic  project 
assignment  by  CNO,  and  consequently  OPTEVFOR  is  often  not 
cognizant  of  the  existence  of  projects  or  their  need  for 
operational  evaluation.   OP-983  publishes  a  quarterly  list 
of  all  assigned  RDT&E  projects  to  be  prosecuted  by  OPTEVFOR, J 
but  no  means  exists  for  ensuring  that  every  project  requiring 
an  operational  evaluation  is  so  assigned.   What  this  means  is 
that  the  Program  Manager  is  able  to  conduct  inadequate  T&E, 
or  he  can  even  obtain  T&E  from  a  number  of  agencies  until  he 
receives  the  results  he  desires;  and  his  actions  may  not  even 
be  criticized. 

Without  questioning  the  integrity  of  the  Navy's 
Program  Managers,  one  can  certainly  speculate  on  the  outcome 
of  a  situation  where  the  Program  Manager  is  typically  facing 
a  cost  growth  combined  with  a  slippage  in  schedule  but  has 
no  clearly  established  requirement  to  obtain  an  "independent" 
test  and  evaluation  of  his  system.   His  likely  course  of 
action  would  be  to  obtain  the  least  expensive  (to  his  program) 
suitable  T&E  that  is  available.   Within  the  current  Navy  T&E 
structure  many  avenues  to  conduct  T&E  are  open  to  the  Program 
Manager:   a  IIAVAIR  T&E  agency,  OPTEVFOR,  or  a  fleet  unit. 
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The  following  is  an  example  which  occurred  in  1970 
in  the  development  of  the  night  carrier  landing  simulator 
for  the  A-7E  airplane.   The  Program  Manager  had  the  fleet 
replacement  A-7  training  squadron  (VA-122)  conduct  the  Navy's 
test  and  evaluation.   The  Naval  Air  Test  Center  and  speci- 
fically the  Carrier  Suitability  Branch,  the  technical  experts 
for  testing  aircraft  and  associated  systems  for  use  in  the 
carrier  environment,  did  not  participate  in  the  test  and 
evaluation  of  the  night  carrier  landing  simulator.   The 
VA-122  T&E  was  certainly  less  expensive  than  would  have  been 
T&E  conducted  by  a  NAVAIR  test  agency.   In  addition  the 
likelihood  of  uncovering  deficiencies  (costly  to  the  Program 
Manager  to  correct)  with  the  simulator  was  far  less  by  having 
untrained  testers  conducting  the  evaluation  than  it  would 
have  been  if  the  critical  voice  of  NATC  had  been  given  the 
opportunity  to  speak.   Although  the  A-7E  night  carrier 
landing  simulator  has  proven  to  be  a  valuable  system,  the 
example  does  illustrate  how  the  Program  Manager  has  the 
latitude  to  bypass  his  own  NAVAIR  test  agencies  as  well  as 
OPTEVPOR. 

Another  illustration  of  the  Acquisition  Manager's 
(the  Program  Manager  for  lower  cost  programs)  latitude  in 
conducting  T&E  is  the  case  of  the  HLU-196A/E  Powered  Bomb 
Hoist.   The  success  of  this  hoist  is  essential  for  the 
implementation  of  the  Improved  Rearming  Rate  Program  aboard 
the  Navy's  attack  carriers.   T&E  of  the  Bomb  Hoist  was 
conducted  by  NATC  and  the  Naval  Weapons  Evaluation  Facility 
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(NWEF)  [Ref.  20].   NATC  recommended  against  clearance  of 
the  HLU-196A/E  for  use  with  all  explosive  ordnance  and  NWEF 
recommended  restriction  against  use  of  the  hoist  with 
nuclear  weapons  until  certain  safety  deficiencies  were 
corrected.   Yet,  the  feature  article  of  the  April  1973 
edition  of  Naval  Aviation  News  [Ref.  21]  reported  that 
carrier  testing  of  the  HLU-I96A/E  had  been  very  successful 
and  implied  that  NATC  and  NWEF  had  obtained  similar  results 
in  their  tests.   Apparently  the  four-day  HLU-196A/E  evalua- 
tion conducted  aboard  USS  Enterprise  during  combat  operations 
by  operational  Navy  ordnancemen  showed  the  HLU-I96A/E  to  be 
very  effective.   The  design  deficiencies  (high  failure  possi- 
bility of  two  critical  springs)  which  NATC  and  NWEF  considered 
to  have  such  an  adverse  effect  on  safety  never  showed  up  in 
this  "operational  evaluation." 

It  appears  that  the  Acquisition  Manager  was  unwilling 
to  accept  the  negative  T&E  results  from  NATC  and  NWEF.   Hope- 
fully, the  serious  design  deficiencies  in  the  HLU-I96A/E  can 
and  will  be  corrected  before  a  catastrophic  accident  occurs. 
One  would  like  to  believe  that  the  acquisition  manager  has 
not  considered  his  system's  design  complete,  merely  because 
someone  was  found  to  report  his  system  as  being  a  "significant 
step  toward  a  mobile  automatic  weapons  loader"  [Ref.  21, 
p.  14],   However,  the  Navy's  acquisition  process  for  non-major 
procurement  provides  an  inadequate  check  to  protect  against 
prematurely  terminating  the  correction  of  deficiencies. 
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7.   Methods  of  Funding 

As  pointed  out  in  Section  IV. A. 2  and  IV. A. 3,  the 
costs  for  conducting  T&E  cannot  be  totally  identified. 
Further  compounding  the  situation  is  the  fact  that  many 
different  methods  are  used  for  funding  the  various  T&E 
agencies.   Some  development  T&E  agencies  are  operated  under 
Navy  Industrial  Funding  where  both  direct  and  indirect  costs 
are  charged  to  each  T&E  project,  and  the  funds  are  then 
transferred  from  the  respective  Program  Manager.   Other 
agencies  utilize  industrial  funding  where  only  direct 
charges  are  incurred  by  the  project,  while  still  others 
never  even  charge  military  personnel  working  directly  on 
the  project  as  a  direct  cost  to  the  project.   Consequently, 
the  cost  to  conduct  T&E  varies  significantly  from  agency  to 
agency. 

In  the  past,  Program  Managers  have  been  forced  by 
monetary  constraints  to  have  certain  tests  conducted  by  a 
T&E  agency  which  was  not  as  well  suited  to  perform  the  T&E, 
but  whose  funding  method  resulted  in  much  lower  cost  for 
the  Program  Manager  and  his  particular  program.   However,  it 
is  quite  possible  that  the  overall  cost  to  the  Navy  might 
have  actually  been  higher,  but  this  is  not  determined  or 
considered  in  the  Program  Manager's  decision  making  process. 
Obviously  a  system  which  forces  a  Program  Manager  to  make 
decisions  based  upon  cost  tradeoffs  must  reflect  realistic 
costs  which,  when  compared,  enable  the  Program  Manager  to 
make  decisions  which  give  valid  cost  savings. 
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The  funding  for  OPTEVFOR  is  different  from  that  of 
the  development  T&E  agencies  in  that  more  funding  sources 
are  involved.   The  VX  Squadrons,  the  Air  Test  and  Evaluation 
Squadrons  that  conduct  aircraft  T&E  for  COMOPTEVFOR,  receive 
part  of  their  funding  from  their  respective  Type  Commanders 
(COMNAVAIRPAC  or  COMNAVAIRLANT) .   Fleet  Commanders  also  fund 


OPTEVFOR  indirectly  through  their  Operating  and  Maintenance 
(O&M)  funds  by  providing  fleet  services  and  support  for  OT&E. 
The  remainder  of  OPTEVFOR  funding  comes  from  the  various 
Systems  Commands  to  support  OT&E  on  a  project  by  project 
basis  [Ref.  15].   As  long  as  the  funding  for  OT&E  is  not 
centrally  managed,  OT&E  will  not  have  fiscal  accountability, 
and  cost-effectiveness  trade-offs  for  conducting  different 
types  of  T&E  will  be  nearly  impossible  to  perform. 
8.   Navy  Test  Range  Management 

Appendix  E  contains  a  list  of  the  major  Navy  ranges 
and  physical  test  facilities  that  are  utilized  in  aircraft 
weapon  system  T&E.   The  ranges  are  controlled  by  a  number  of 
different  commands.   The  only  central  range  authority  is  the 
Naval  Material  Command  (NAVMAT)  Navy  Range  Manager,  who  only 
has  line  management  control  over  the  NAVMAT  and  NAVAIRSYSCOM 
controlled  ranges.   It  is  significant  to  note  that  OPTEVFOR 
does  not  have  control  over  any  range  or  physical  test 
facility.   Consequently,  OT&E  is  conducted  at  ranges  and 
facilities  which  are  primarily  development  T&E  oriented. 
The  tests  are  planned  and  conducted  around  the  specific  range 
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or  facility  requirements,  rather  than  on  the  basis  of  what 
is  required  for  optimal  OT&E. 

The  cost  to  operate  the  ranges  and  physical  facilities 
cannot  be  isolated,  since  they  are  buried  in  the  budgets  of 
the  various  controlling  agencies.   Without  central  range 
management  or  knowledge  of  the  overall  operating  costs,  it 
is  a  near  impossible  task  to  justify  the  various  ranges  and 
make  rational  decisions  when  forced  to  reduce  range  assets. 
Since  the  ranges  available  for  military  use  determine  the 
location  and  the  functions  of  the  various  T&E  facilities, 
this  area  is  critical  to  T&E.   During  this  period  of  increased 
civilian  interest  in  the  vast  amount  of  real  estate  controlled 
by  the  military,  it  behooves  the  Navy  to  devote  its  best 
efforts  to  ensure  that  adequate  ranges  and  facilities  will 
exist  for  T&E  in  the  future. 

The  under-utilization  of  many  test  ranges  makes  them 
candidates  for  elimination  or  conversion  to  some  other  use. 
The  General  Services  Administration  is  conducting  a  continual 
survey  to  identify  under-utilized  and/or  poorly  utilized  real 
property  holdings  of  all  Government  agencies  [Ref.  22].   The 
Navy  must  determine  its  total  Navy  Range  requirements,  and 
then  determine  if  it  has  an  excess  or  a  shortage.   With  so 
many  Navy  agencies  controlling  ranges  and  facilities,  it 
becomes  a  difficult  task  to  separate  the  actual  Navy  range 
requirements  from  the  internally  generated  requirements  where 
each  agency  tries  to  "keep  what  it  has." 
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B.   OTHER  DEFICIENCIES 

Besides  the  organizational  features  of  the  Navy's  T&E 
process  many  other  phenomena  exist  which  cause  the  Navy's 
T&E  process  to  be  deficient.   This  section  will  discuss 
these  other  deficiencies: 

(1)  Overlap  between  DT&E  and  OT&E 

(2)  Inadequate  Test  Planning 

(3)  Inadequate  operational  test  objectives 

(4)  Inadequate  technical  expertise  in  OPTEVFOR 

(5)  Lack  of  a  Weapon  System  Test  and  Evaluation  School  * 

(6)  Late  test  results 

(7)  BIS  scheduled  prior  to  DSARC  III 

(8)  Shortage  of  analysts 

(9)  T&E  not  always  treated  as  a  continuing  process 

1.   Overlap  Between  OT&E  and  DT&E 

Perhaps  the  greatest  problem  facing  military  T&E 
today  is  the  lack  of  a  uniform  understanding  of  the  differ- 
ences between  OT&E  and  DT&E.   The  definition  of  OT&E  and 
DT&E  presented  in  Appendix  B  may  seem  quite  explicit  to  the 
uninitiated.   The  consensus  throughout  the  Navy's  aircraft 
T&E  community,  however,  is  that  the  major  portion  of  T&E 
conducted  by  the  military  falls  somewhere  in  a  "grey  area" 
that  is  not  purely  DT&E  nor  purely  OT&E. 

Since  the  majority  of  T&E,  therefore,  is  conducted 
in  an  area  that  is  not  clearly  the  responsibility  of  the 
DT&E  agency  or  the  OT&E  agency,  parochial  attitudes  are 
typical  and  often  hinder  cooperative  testing.   Each  agency 
attempts  to  lay  claim  to  the  larger  or  certainly  the  more 
desirable  portions  of  the  system's  particular  testing 
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equirements.   There  exists  a  definate  tendency  for  opera- 
ional  test  pilots  to  want  to  become  more  involved  with  the 
ore  glamorous  experimental  or  developmental  testing, 
ecently,  more  and  more  pilots  with  only  VX/operational  T&E 

ckgrounds  have  been  accepted  for  membership  in  the  Society 
f  Experimental  Test  Pilots,  an  organization  which  requires 
Ktensive  experience  in  developmental  or  experimental  flight 
*sting  for  full  membership.   These  VX  pilots  have  acquired 
le  requisite  experimental  and  developmental  test  pilot 
xperience,  but  they  did  it  while  in  operational  test  pilot 
Lllets. 

A  similar  tendency  exists  on  the  part  of  the  develop- 
mt  test  agencies.   The  development  test  pilot/NFO  (Naval 
Light  Officer)  resents  being  put  into  the  current  situation 
lere  he  feels  that  he  is  asked  only  for  technical  opinions, 
id  OPTEVFOR  is  relied  upon  for  operational  opinions.   The 
jvelopmental  T&E  project  officers  have  operational  back- 

ounds  comparable  with  those  of  the  operational  T&E  project 
'fleers  in  OPTEVFOR.   The  only  significant  difference  is 
lat  the  DT&E  project  officers  spend  a  year  in  Test  Pilot 
:hool  after  leaving  their  last  operational  billet  and  prior 

being  assigned  to  a  DT&E  agency.   Consequently,  the  DT&E 
'oject  officers  adamantly  refuse  to  accept  roles  that  are 
Jolated  from  operational  suitability,  and  they  tailor  their 
ssts  to  look  at  the  operational  aspects  as  well  as  the 
:chnical  aspects  of  a  weapons  system. 
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This  cross-pollination  of  mission  roles  between  the 
DT&E  agencies  and  OPTEVFOR  probably  causes  each  side  to 
become  better  "military  T&E"  organizations.   However,  the 
U.  S.  Navy  cannot  afford  the  luxury  of  developing  this 
expensive  duplication.   As  long  as  separate  agencies  are 
being  tasked  to  perform  T&E  which  does  not  lend  itself  to 
an  easy  separation  into  DT&E  and  OT&E,  parochialism  will 
continue  to  grow  at  the  expense  of  T&E  efficiency  in  terms 
of  cost  and  performance . 

2.   Inadequate  Test  Planning 

DOD  Directives  5000.1  and  5000.3  require  early 
planning  for  T&E.   Prior  to  the  DSARC  I  Program  Initiation 
decision  the  critical  questions  and  areas  of  risk  that  have 
to  be  resolved  by  T&E  must  be  identified.   At  this  time  the 
test  objectives,  schedules,  and  milestones  must  also  be 
determined.   This  can  only  be  accomplished  through  extensive 
early  test  planning  with  the  assistance  of  the  T&E  agencies. 

Unfortunately,  the  T&E  agencies  are  not  involved  in 
the  early  stages  of  test  planning.   A  Test  and  Evaluation 
Master  Plan  (TEMP)  is  required  to  be  prepared  as  early  as 
possible  in  the  acquisition  cycle.   It  is  a  management  docu- 
ment which  describes  how  and  when  developmental  and  opera- 
tional test  objectives  will  be  met,  and  its  preparation  Is 
the  responsibility  of  the  development  agency's  Program 
Manager  [Ref.  11].   The  T&E  agencies  become  involved  only 
through  a  review  and  comment  process  rather  than  being 
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intimately  involved  in  determining  test  objectives  and  how 
best  to  determine  if  they  are  met . 


n 


To  translate  this  situation  into  the  commercial 
DOD  market  place,  when  DOD  procures  something  for  a  contrac- 
tor, the  user,  in  this  case  DOD,  tells  the  contractor  or 
producer  what  tests  will  be  performed  to  demonstrate  the 
acceptability  of  the  particular  item.   The  contractor  may 
assist  in  the  structuring  of  the  tests  or  in  recommending 
substitute  tests,  but  the  responsibility  for  test  planning 
to  determine  service  suitability  clearly  must  remain  with 
DOD.   In  going  back  to  the  Navy's  internal  situation,  a 
similar  situation  should  but  does  not  exist.   The  producer, 
in  this  case  the  developing  agency,  has  the  responsibility 
of  preparing  the  test  plan  for  each  particular  system.   The 
user,  in  this  case  OPNAV,  merely  assumes  a  review  and  comment 
role. 

Certainly  the  Program  Manager  must  retain  the 
responsibility  for  test  scheduling  and  for  satisfactorily 
meeting  the  test  objectives.   However,  the  user  and  the 
T&E  experts  should  have  the  primary  responsibility  for  the 
determination  of  the  test  objectives  and  how,  when,  where, 
and  by  whom  they  can  best  be  tested  to  determine  if  they  are 
being  met  satisfactorily. 

3.   Inadequate  Operational  Test  Objectives 

The  preceding  section  discussed  the  inadequacy  of 
test  planning  in  the  early  stages  of  a  system's  development. 
This  has  a  most  adverse  effect  on  OT&E.   Very  frequently 
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OPTEVFOR  is  tasked  to  perform  an  "operational  evaluation" 
on  a  system  or  subsystem,  but  a  standard  against  which  to 
test  does  not  exist.   Unfortunately,  the  operational  require- 
ments for  many  weapon  systems  are  never  well  promulgated  nor 
explicitly  defined. 

The  Specific  Operational  Requirement  (SOR)  is  the 
requirement  document  from  CNO  to  which  a  system  is  designed. 
The  SOR  is  supposed  to  define  the  required  weapon  system 
capability  in  terms  of  mission  requirements,  operational 
concept,  and  performance  constraints  [Ref.  23].   It  would 
appear  that  the  SOR  would  be  the  "standard"  against  which 
OPTEVFOR  could  design  its  tests.   In  reality,  the  SOR  is 
often  incomplete  or  outdated  by  the  time  the  system  is  ready 
for  any  T&E,  or  the  SOR  has  not  been  written  and  approved 
by  the  time  T&E  has  commenced.   One  further  deficiency  is 
that  the  SOR  usually  contains  little  in  the  way  of  test 
planning  or  test  objectives  —  that  is  left  entirely  to  the 
developing  agency  in  its  response  to  the  SOR,  the  Technical 
Development  Plan  (TDP). 

The  TDP  is  the  developing  agency's  management  docu- 
mentation of  its  proposed  actions  and  procedures,  and  the 
resources  which  are  required  in  order  to  achieve  the  capa- 
bility described  in  the  SOR.   The  TDP  is  a  primary  planning 
document  for  the  development,  production,  installation, 
integrated  logistic  support,  reliability  and  maintainability, 
personnel  training,  and  test  and  evaluation  of  the  weapon 
system  [Ref.  23].   The  in-depth  analysis  that  goes  into  the 
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TDP  translates  the  requirements  of  the  SOR  (if  one  exists) 
into  specific  performance  and  development  objectives 
against  which  DT&E  agencies  can  test.   However,  the  TDP  is 
not  the  document  against  which  OPTEVFOR  should  be  testing. 

A  comprehensive  analysis  should  be  conducted  early 
in  the  requirements  determination  phase  (Conceptual  Phase) 
to  develop  operational  effectiveness  criteria.   These  cri- 
teria would   serve  the  valuable  function  for  establishing 
a  general  method  to  evaluate  and  assess  whether  the  system's 
development  is  meeting  the  operational  needs.   Operational 
effectiveness  criteria  should  facilitate  the  writing  of  a 
flexible  test  plan  which  is  required  by  OT&E.   The  SOR  or  a 
suitable  replacement  document  should  contain  such  operational 
effectiveness  criteria.   This  is  clearly  the  role  of  the  user, 
i.e.,  OPNAV,  and  it  should  not  be  delegated  to  the  developing 
agency . 

4.   Inadequate  Technical  Expertise  in  OPTEVFOR 

There  appears  to  be  a  common  belief  that  the  main 
requirement  for  an  OT&E  project  officer  is  that  he  possess 
current  fleet  operational  experience.  While  extensive 
operational  experience  is  a  necessity  for  the  molding  of  a 
competent  OT&E  project  officer,  his  training  should  also 
include  schooling  in  the  technical  disciplines  that  are 
involved  in  his  project;  aeronautics,  electronics,  physics, 
and  test  methods  (including  statistics  and  test  design). 

It  is  a  well  established  requirement  that  the 
majority  of  the  project  officers  (pilots  and  Naval  Flight 
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Officers)  involved  with  DT&E  are  graduates  of  a  formal  Test 
Pilot  School  where  the  discipline  of  T&E  is  taught  as  are 
also  many  basic  technical  fundamentals.   In  addition  each 
DT&E  agency  has  an  extensive  civilian  technical  work  force 
which  is  composed  of  aeronautical,  mechanical,  and  electrical 
engineers,  mathematicians,  physicists,  and  computer  special- 
ists.  An  in-depth  technical  background  has  been  recognized 
by  the  DT&E  agencies  as  an  absolute  requirement  for  con- 
ducting T&E  of  a  sophisticated  weapon  system. 

Many  OPTEVFOR  project  officers  believe  that  opera- 
tional test  objectivity  is  decreased  as  technical  expertise 
is  increased.   However,  the  increased  T&E  role  that  OPTEVFOR 
is  being  forced  to  fulfill  is  causing  OPTEVFOR  to  recognize 
the  need  to  increase  its  level  of  technical  expertise.   The 
number  of  project  officers  possessing  both  operational 
experience  and  technical  capability  must  be  expanded  if 
OPTEVFOR  is  going  to  fulfill  adequately  its  T&E  objectives 
while  efficiently  utilizing  test  resources. 

By  designing  efficient  test  plans,  millions  of 
dollars  can  often  be  saved,  while  obtaining  even  more 
conclusive  results.   The  design  of  such  test  plans  cannot 
be  accomplished  by  personnel  possessing  only  a  primary 
attribute  of  recent  operational  experience.   Operational 
testing  should  be  conducted  by  personnel  having  similar 
qualifications  to  those  who  will  ultimately  utilize  the 
system,  but  this  should  not  preclude  personnel  from  possessing 
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the  requisite  technical  capabilities  from  designing  and 
planning  the  tests  and  analyzing  and  evaluating  the  results. 

OPTEVFOR  recently  evaluated  a  system  in  USS  Ranger. 
The  system's  performance  was  believed  to  be  affected  by 
latitude,  longitude,  look  angle,  and  other  parameters. 
Testing  for  latitude  and  longitude  effects  would  necessitate 
relocating  the  test  force  all  over  the  world.   Fortunately 
a  project  officer  assigned  to  this  program  had  an  excellent 
technical  background.   He  was  able  to  show  by  some  relatively 
simple  preliminary  tests  that  both  latitude  and  longitude 
had  an  insignificant  effect  on  one  performance  parameter  in 
question.   Latitude  and  longitude  would  affect  this  parameter 
only  as  a  result  of  the  consequential  changes  in  look  angle. 
This  knowledge  could  provide  considerable  cost  savings,  since 
only  look  angle  must  be  varied  to  complete  this  portion  of 
the  test.   One  can  only  wonder  how  many  tests  consist  of 
expensive  testing  for  the  effects  of  parameters  which  could 
have  been  shown  to  be  insignificant  by  a  skilled  analyst. 

Another  area  where  OPTEVFOR' s  technical  expertise  is 
lacking  is  in  the  testing  of  computer  programs  (software), 
which  have  become  an  integral  part  of  every  current  weapon 
system.   The  results  of  inadequately  designed  tests  of  these 
weapon  systems  will  not  show  where  system  problems  exist, 
i.e.  in  the  computer  program,  in  the  hardware,  in  the  operating 
procedures,  etc.   If  one  is  going  to  expend  the  effort  and  cost 
to  conduct  the  tests,  it  is  only  logical  to  expect  that  the 
results  will  be  able  to  show  where  problems  exist,  not  merely 
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that  the  system  operation  is  good  or  bad.   In  order  to 
design  adequate  operational  or  developmental  tests  for 
computer  software,  an  in-depth  knowledge  of  the  system 
operation  is  required,  as  is  also  a  good  basic  understanding 
of  the  technical  principles  Involved. 

Technical  expertise  in  OPTEVFOR  can  be  increased  by 
obtaining  more  civilian  analysts  and  other  technical  special- 
ists and  by  increasing  the  formal  training  given  to  the 
project  officers.   The  assignment  of  only  one  civilian,  an 
analyst,  to  each  OPTEVFOR  VX  squadron  is  an  inadequate  core 
of  technical  expertise  around  which  the  VX  squadron  can 
perform  efficiently  and  effectively.   The  recent  efforts  by 
OPTEVFOR  to  obtain  more  officers  with  graduate  technical 
degrees  from  the  Naval  Postgraduate  School  is  a  move  in  the 
right  direction,  but  much  more  emphasis  has  to  be  given  to 
this  area. 

5.   Lack  of  a  Weapon  System  T&E  School 

Very  little  formal  training  is  required  or  is  given 
before  project  officers  are  assigned  to  an  OPTEVFOR  project. 
VX-*!  project  officers  are  probably  the  most  formally  trained 
in  OPTEVFOR.   They  usually  attend  a  two  week  training 
program  at  Deputy  COMOPTEVFOR,  North  Island,  California  and 
then  a  formal  three  day  training  course  on  the  operation  and 
utilization  of  the  Pacific  Missile  Range  (PMR),  their  primary 
test  site.   But  the  main  part  of  every  OPTEVFOR  project 
officer's  training  is  "on-the-job  training"  —  also  called 
"learning  from  one's  own  mistakes."   In  the  T&E  environment 
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mistakes  are  too  costly  a  teaching  tool.   It  costs  approxi- 
mately $36,000  per  hour  to  utilize  the  PMR.   This  should 
compel  any  user  to  ensure  that  his  tests  are  conducted  as 
efficiently  and  expeditiously  as  possible  with  minimal 
likelihood  of  having  to  repeat. 

Although  the  majority  of  DT&E  agency  project  officers 
have  attended  a  formal  one  year  course  of  study  at  a  Test 
Pilot  School,  the  knowledge  they  receive  is  primarily  in  the 
discipline  of  aircraft  flying  qualities  and  performance  T&E. 
While  this  is  essential  to  anyone  becoming  a  project  officer 
in  the  Plight  Test  Division  of  NATC,  approximately  two-thirds 
of  the  U.  S.  Navy  Test  Pilot  School  graduates  go  to  project 
officer  jobs  where  there  is  very  little  need  to  utilize  any 
of  the  flying  quality  and  performance  test  techniques.   What 
is  lacking  in  the  training  of  these  project  officers  is 
extensive  training  in  the  T&E  of  complex  weapons  systems. 
The  U.  S.  Navy  Test  Pilot  School  has  introduced  a  course 
entitled  Integrated  Weapons  Systems,  but  it  composes  only 
7  percent  of  the  formal  classroom  instruction  time  and  serves 
the  purpose  of  giving  the  student  only  an  introduction  into 
the  complex  field  of  weapon  system  T&E  (as  opposed  to  aircraft 
vehicle  T&E) . 

As  with  OPTEVFOR  "on-the-job  training"  then  becomes 
the  primary  training  method  for  the  DT&E  project  officer  who 
becomes  involved  with  weapon  system  testing.   But  the  DT&E 
project  officer  has  many  distinct  advantages  over  his  OPTEVFOR 
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counterpart.   The  Test  Pilot  School  has  given  the  DT&E 
project  officer  considerable  knowledge  and  experience  in 
the  field  of  T&E  (i.e.,  how  to  plan,  conduct,  and  analyze 
tests  efficiently),  and  it  has  also  taught  him  how  to  write 
concise,  accurate  test  reports. 

DT&E  project  officers  receive  more  formal  training 
than  their  OPTEVFOR  counterparts.   However,  a  significant 


deficiency  exists  in  the  training  of  both  types  of  project 
officers.   Nowhere  are  they  taught  how  to  test  a  complex 

weapon  system,  how  to  separate  computer  software  problems 

f 

from  system  hardware  problems,  how  to  design  an  optimal  test 

using  a  statistical  design  of  experiments  approach,  etc. 

Not  only  do  the  project  officers  receive  inadequate 
formal  training,  but  so  do  the  civilian  engineers  throughout 
the  T&E  community.   Most  of  the  civilian  engineers  working 
in  the  Flight  Test  Division  of  NATC  complete  the  entire 
academic  curriculum  of  the  Navy  Test  Pilot  School.   However, 
formal  training  in  T&E  does  not  exist  for  civilian  engineers 
at  other  T&E  agencies.   Although  the  civilians  possess 
technical  or  engineering  degrees,  very  few  civilians 
reporting  to  a  T&E  agency  have  had  any  prior  experience  or 
training  in  the  discipline  of  T&E.   Consequently,  costly 
on-the-job  training  in  T&E  is  again  the  technique  utilized  to 
train  the  civilian  engineers  and  analysts. 
6 .   Late  Test  Results' 

The  new  DOD  T&E  policies  are  intended  to  correct  the 
deficiency  of  T&E  being  conducted  too  late  to  affect  the 
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system's  development.   However,  Program  Managers  are  still 
plagued  with  the  problem  of  not  getting  test  results  soon 
enough  from  the  test  agencies  which  they  have  requested  to 
conduct  certain  tests.   This  is  usually  not  a  problem  with 
NPE  or  BIS  reports  where  the  schedule  for  report  submission 
is  very  rigid,  and  all  T&E  agencies  are  forced  to  comply. 
The  problem  arises  with  the  myriad  of  minor  T&E  projects 
that  are  conducted  during  every  major  weapon  system's 
development.   The  test  reports  from  the  DT&E  agencies  go 
through  such  a  lengthy  review  process  that  they  are  frequently 
received  by  the  Program  Manager  too  late  to  assist  in  his 
decision  making  process.   Figure  5  shows  a  typical  DT&E  agency 
test  report  review  chain. 

Not  every  test  report  is  required  to  follow  the 
complete  review  chain,  but  all  of  the  test  reports  dealing 
with  important  projects  do  follow  it.   Some  type  of  formal 
review  chain  is  an  absolute  requirement  to  ensure  that  the 
test  agency's  main  output,  test  and  evaluation  reports,  do 
not  fall  below  the  established  minimum  level  of  quality, 
and  to  ensure  that  misleading  statements  or  statements  not 
supported  by  data  are  not  made  in  the  reports.   However, 
when  the  review  chain  becomes  so  time  consuming  that  it 
becomes  the  major  obstacle  in  the  conduct  of  T&E,  a  signi- 
ficant change  should  be  made . 

7.   BIS  Scheduled  Prior  to  DSARC  III 

OPNAVINST  3960.8  shows  BIS  commencing  along  with 
OPEVAL  prior  to  the  DSARC  III  major  production  decision. 
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NATC  Staff  Review 
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Division  Director  and  Asst.  Director 
review  —  2  people 


Division  Chief  Project  Officer  Review 
2  to  3  people 


Project  Officer  internal  Branch  Review 
3  to  k   people 


Project  Officer  and  Project  Team  Conduct 
tests  and  write  report 


Typical  DT&E  Agency  Test  Report  Review  Chain 

FIGURE  5 


69 


(See  Figure  2).   The  function  of  the  OPEVAL  is  to  determine 
if  the  system  is  ready  for  production,  and  the  function  of 
BIS  is  to  determine  if  the  production  aircraft  are  acceptable 
for  service  use  and  whether  the  system's  contract  specifica- 
tions have  been  met  [Refs.  2  and  20], 

If  this  instruction  is  to  be  followed  in  regard  to 
the  time  sequencing  of  BIS  and  OPEVAL,  the  functions  of  BIS 
aircraft  trials  will  have  to  be  modified.   However,  if  it  is 
modified  so  as  to  fulfill  a  function  prior  to  the  major 
production  decision,  this  function  would  obviously  have  to 
be  related  to  determining  whether  a  favorable  production 
decision  should  be  made.   As  discussed  previously  (Section 
IV. A. 3),  the  roles  of  BIS  and  OPTEVFOR  are  too  similar  at 
present,  and  such  a  change  to  the  function  and  objectives 
of  BIS  would  exacerbate  the  duplication  that  already  exists 
between  the  function  of  BIS  and  the  functions  of  OPEVAL. 
The  other  alternative  is  to  have  BIS  remain  as  service 
acceptance  trials  and  have  it  occur  after  OPEVAL,  which  is 
a  radical  change  from  the  current  sequence  of  aircraft  T&E. 
;8.   Shortage  of  Analysts 

The  requirement  to  utilize  statistical  analysts  for 


designing  tests  and  analyzing  the  data  for  statistical 
significance  has  long  been  known  and  practiced  by  OPTEVFOR. 
The'  DT&E  agencies  are  still  living  in  a  world  where  they 
believe  "everything"  is  deterministic,  and  very  seldom  is 
the  statistical  aspect  of  data  fully  analyzed.   This  is 
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typical  of  the  entire  scientific  and  engineering  community. 
Only  within  the  last  five  to  ten  years  has  emphasis  been 
placed  upon  the  probabilistic  aspects  of  engineering. 

The  DT&E  agencies  need  to  devote  more  attention  to 
the  investigation  of  the  statistical  aspects  of  their  tests. 
A  large  amount  of  DT&E  testing  is  done  where  statistics 
(i.e.  means,  variances,  confidence  levels)  are  calculated, 
but  a  statistical  analyst  i'sseldom  if  ever  used  to  verify 
the  credibility  of  the  test.   While  a  large  part  of  "our 
world"  might  be  distributed  normally,   in  many  instances 
this  assumption  can  lead  to  some  significant  errors  in  the 
conclusions  one  draws  from  data. 

Although  OPTEVPOR  has  recognized  the  need  for 
analysts,  it  still  has  serious  T&E  problems  that  can  be 
traced  to  the  lack  of  sufficient  analysis  expertise.   There 
is  presently  one  civilian  analyst  assigned  to  each  VX 
squadron  and  four  military  and  four  civilian  analysts  (two 
are  permanent  Government  Service  employees)  assigned  to 
OPTEVFOR  Headquarters  staff  [Ref.  14],   The  problem  with  the 
analysts  is  twofold:   (1)  low  experience  level,  and 
(2)  inadequate  numbers  of  analysts  to  perform  all  required 
functions. 


Populations  of  random  variables  are  distributed  according 
to  different  types  of  statistical  distributions.   The  Gaussian 
or  Normal  distribution  is  one  of  the  most  common  and  a  great 
many  test  parameters  and  conditions  can  accurately  be 
described  by  the  Normal  distribution.   However,  normality 
should  always  be  tested  for  and  not  merely  assumed. 
Incorrectly  assuming  a  Normal  distribution  when  some  other 
distribution  is  appropriate  could  lead  to  erroneous  conclusions 
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The  civilian  analysts  assigned  to  the  VX  squadrons 
and  two  assigned  to  OPTEVFOR  headquarters  work  for  the 
Operations  Evaluation  Group  (OEG),  a  part  of  the  Center  for 
Naval  Analysis  (CNA) .   These  OEG  representatives  are  assigned 
to  OPTEVFOR  for  a  two  year  period  (recently  increased  from 
only  one  year).   For  the  OEG  representative  this  two  year 
period  is  his  field  experience  tour.   Generally,  the  OEG 
personnel  are  relatively  junior  when  they  are  assigned  to 
OPTEVFOR,  and  consequently  a  large  amount  of  their  time  is 
spent  in  on-the-job  training.   The  best  career  promotion 
opportunity  for  the  OEG  personnel  appears  to  exist  when  they 
are  assigned  to  jobs  within  CNA  headquarters.   Consequently, 
there  is  little  motivation  for  an  OEG  representative  to 
continue  working  for  OPTEVFOR  despite  the  fact  that  he  may 
find  his  job  challenging,  rewarding,  and  essetnial.   As  a 
result  of  these  circumstances,  the  experience  level  and 
caliber  of  the  OEG  representatives  assigned  to  OPTEVFOR  is 
less  than  desired. 

The  shortage  of  analysts  precludes  utilizing  an 
analyst's  expertise  to  design  each  of  the  test  plans; 
although  the  analyst  may  try  to  review  each  test  plan  prior 
to  its  execution.   In  practice  the  majority  of  OPTEVFOR 
test  plans  are  constructed  by  going  through  the  files  and 
modifying/copying  a  previously  executed  test  plan.   As  a 
result,  many  tests  are  inadequately  planned. 

Many  times  the  analyst  is  not  consulted  until  after 
the  test  has  been  completed,  and  he  is  then  asked  to 
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determine  what  the  data  show.   This  is  inadequate,  since 
the  test  has  to  be  designed  to  obtain  data  in  a  fashion 
that  will  permit  data  analysis  from  which  valid  conclusions 
can  be  drawn.   However,  even  OPTEVFOR's  Project  Instruction 
Volume  II  [Ref.  24],  which  is  a  comprehensive  handbook  for 
conducting  OT&E  and  analyzing  the  data,  essentially  overlooks 
this  essential  part  of  T&E,  the  design  of  the  experiment. 
As  a  result  of  the  lack  of  an  adequate  number  of 
capable  analysts  to  design  the  tests  and  then  analyze  the 
data,  needless  expenses  are  often  incurred,  and  incorrect 
conclusions  can  easily  result.   An  example  of  needless 
expense  is  the  recent  OPTEVFOR  testing  of  a  missile  which 
was  supposed  to  demonstrate  a  kill  probability  of  0.4. 
Fifteen  missiles  were  obtained  to  conduct  the  tests  and  all 
fifteen  were  fired.   The  test  personnel  reasoned  that  if  six 
or  more  "hits"  were  obtained,  a  kill  probability  of  0.4  or 
greater  would  have  been  demonstrated.   (6  hits/15  Missile 
Firings  =  Hit  or  Kill  Probability  Pk  =  0.4.)   A  simple 
sequential  test  could  have  been  designed  where  the  firing 
of  each  missile  was  dependent  upon  the  results  of  the 
previous  firings.   After  each  firing  one  of  the  following 
decisions  could  be  made:   accept  the  hypothesis  that  Pk  is 
greater  than  or  equal  to  0.4,  reject  the  hypothesis,  or  fire 
another  missile.   As  an  illustration  of  this  test  plan  assume 
the  first  six  missiles  resulted  in  misses.   This  would 
demonstrate  at  a  95  percent  confidence  level  that  the  kill 
probability  is  less  than  the  required  0.4.   This  result  is 
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obtained  by  using  the  binomial  distribution  with  six  failures 
out  of  six  attempts  and  a  test  hypothesis  that  Pk  is  greater 
than  or  equal  to  0.4.   In  this  case,  it  would  be  a  needless 
expense  to  fire  any  more  than  six  missiles.   If  all  fifteen 
missiles  were  fired,  the  last  nine  would  essentially  be 
wasted,  since  no  greater  statistical  significance  would  need 
to  be  achieved.   The  actual  missile  test  resulted  in  some 
"hits,"  but  more  missiles  were  fired  than  was  necessary  to 
reject  the  test  hypothesis  that  the  Pk  was  greater  than  or 
equal  to  0.4. 

9.   T&E,  Not  Always  Treated  as  a  Continuing  Process 

In  order  for  T&E  results  to  be  used  to  accurately 
predict  or  assess  a  weapon  system's  operational  effective- 
ness, continuous  monitoring  of  the  system's  development 
progress  must  be  maintained.   Snapshot  looks  at  the  weapon 
system  at  infrequent  intervals  will  not  give  a  valid  assess- 
ment of  a  system's  potential. 

Making  accurate  assessments  of  a  system's  military 
worth  and  operational  effectiveness  requires  considerable 
T&E  experience  on  the  part  of  the  evaluator.   As  an  illustra- 
tion of  the  need  to  monitor  continually  the  development 
progress  of  a  weapon  system,  this  section  will  briefly 
describe  the  availability  growth  that  should  be  tracked  in 
every  development  program. 

Operational  Availability  (Ao)  is  defined  as  the 
ratio  of  the  total  time  the  system  is  capable  of  performing 
its  function  (uptime)  to  the  total  time  when  there  is  a 
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demand  for  the  system  (uptime  plus  total  down  time  which 
includes  maintenance/logistic  delay  times).   Availability 
can  also  be  expressed  as  "the  probability  that  the  system 
will  operate  satisfactorily  at  any  point  in  time  when  used 
under  stated  conditions"  [Ref.  25]. 

Mathematically  operational  availability  is  defined 
as  follows: 


.         MTBM 
Ao  = 


MTBF  +  MDT 


where:      MTBM  =  mean-time-between-maintenance 

(includes  corrective  and  preventative 
maintenance ) 

MDT  =  mean-down-time  including  maintenance 
and  logistic  delay  times 


Figure  6  is  an  illustrative  time  history  of  a  system's 
operational  availability  growth  during  its  development  and 
initial  deployment.   Typically ,  the  operational  availability 
will  not  be  a  linear  function. 

The  evaluator  needs  considerable  experience  to 
accurately  interpret  such  growth  time  histories.   In  the 
included  example  the  dip  in  the  curve  after  commencement  of 
military  T&E  is  typical  and  is  usually  caused  by  the 
problems  attendant  with  the  new  operator  developing  his 
learning  curve  in  operating  the  system.   The  learning  curve 
phenomenon  occurs  again  after  the  system  is  introduced  to 
the  fleet  and  a  similar  dip  in  operational  availability  is 
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Is  usually  encountered.   In  this  case  increased  system 
familiarity  by  the  fleet  results  in  more  critical  demands 
on  system  performance  and  consequently  more  deficiencies/ 
failures  in  the  system  will  be  detected.   The  rapid  usage 
of  the  spares  reduces  the  level  of  spares  to  the  point  that 
it  adversely  affects  operational  availability.   This  example 
illustrates  why  frequent  tracking  of  operational  availability 
rather  than  occasional  snapshot  looks  is  required  to  accurately 
evaluate  a  system's  operational  availability  growth.   This 
also  applies  to  the  tracking  of  a  system's  other  operational 
suitability  and  performance  parameters. 

C.   GOOD  FEATURES  OF  THE  NAVAL  AIRCRAFT  T&E  PROCESS 

Although  numerous  deficiencies  exist  in  the  Navy's  T&E 
process  for  aircraft  weapon  systems,  many  good  features 
exist: 

(1)  Navy  test  pilots  and  NFO's  (Naval  Flight  Officers) 
typically  have  and  maintain  excellent  operational 
backgrounds 

(2)  Testing  is  conducted  by  relatively  small  organizations 

(3)  Competition  among  test  agencies  stimulates  good 
performance 

(*!)  Decentralized  test  management  is  practiced 

(5)  T&E  attracts  top  caliber  military  personnel. 

The  Navy's  developmental  and  operational  test  pilots  and 

NFO's  are  required  to  have  recent  operational  experience 

before    receiving  orders   to   a  T&E   project   officer's   billet. 

Pilots  and  NFO's  are  selected  for  admission  to  the  Navy  Test 

Pilot  School  on  the  basis  of  their  operational,  academic, 

and  flight  backgrounds.   After  graduation  the  engineering 
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test  pilot  or  test  flight  officer  is  assigned  to  a  project 
officer's  billet  in  one  of  the  T&E  Divisions  at  NATC,  or 
occasionally  NMC.   Within  three  years  the  pilot  or  NFO  is 
ordered  back  to  an  operational  billet.   In  this  way,  the 
Navy  keeps  bringing  current  operational  experience  into  its 
DT&E  community,  and  test  pilots  do  not  remain  in  test 
billets  for  extended  periods  of  time,  so  that  they  do  not 
lose  their  operational  perspective.   The  Air  Force  operates 
considerably  different  in  that  many  of  their  test  pilots 
spend  the  majority  of  their  flying  careers  assigned  to  T&E 
billets,  and  they  do  not  get  re-exposure  to  operational 
environments.   (The  Viet  Nam  crisis  somewhat  changed  this 
in  that  most  Air  Force  test  pilots  were  given  one  year  tours 
of  duty  in  Southeast  Asia) . 

Navy  aircraft  T&E  is  conducted  by  relatively  small 
organizations  where  the  project  officer  has  ready  access  to 
the  Commander  of  the  test  activity.   The  so  called  "can  do" 
attitude  is  evident  in  each  of  the  T&E  activities.   This 
attitude  is  generally  easier  to  develop  in  a  small  organiza- 
tion than  in  a  larger  bureaucratic  organization. 

Since  many  T&E  activities  have  similar  capabilities, 
competition  is  generated  between  these  activities  in  trying 
to  solicit  more  project  work,  obtain  greater  test  capabili- 
ties, -etc.   This  competition  assists  in  causing  each  activity 
to  become  more  cohesive,  more  work  and  task  oriented,  and 
better  organized.   Loyalty  and  pride  in  one's  test  activity 
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is  prevalent  and  further  facilitates  good  performance  from 
a  test  activity. 

The  project  officer  and  his  team  plan  and  conduct 
individual  tests.   High  level  T&E  management  develops  and 
approves  the  TEMP  and  other  broad  T&E  guidelines,  but  within 
these  bounds  the  project  officer  is  given  essentially  a  free 
hand  in  the  development  of  his  test  plan.   This  decentralized 
management  in  planning  and  conducting  the  tests  causes  the 
personnel  conducting  the  test  to  be  more  responsible  for  the 
timely  and  effective  accomplishment  of  the  test  objective. 

The  Navy  has  made  aircraft  T&E  a  challenging,  interesting, 
and  rewarding  job  assignment.   Consequently,  the  Navy  has 
been  able  to  constantly  attract  the  highest  caliber  personnel 
for  jobs  in  the  DT&E  agencies  and  OPTEVFOR.   A  tour  of  duty 
in  a  T&E  Agency  is  regarded  throughout  the  Naval  Aviation 
community  as  career  enhancing,  and  this  further  motivates 
good  people  to  seek  tours  of  duty  in  T&E. 

D.   A  PREDICTION  OF  THE  EVOLUTION  OP  T&E  BASED  ON  NO 
MAJOR  CHANGES  TO  THE  CURRENT  T&E  PROCESS 

Unless  major  changes  are  made  to  the  Navy's  current  T&E 

process,  a  gradual  and  costly  evolution  will  occur,  and  the 

roles  of  the  T&E  agencies  will  be  altered  significantly.   A 

basic  premise  of  this  thesis  is  that  if  the  Navy  can  define 

an  optimal  T&E  process,  it  should  make  whatever  changes  are 

necessary  to  achieve  this  process,  rather  than  relying  on  a 

slow  evolutionary  process  which  may  or  may  not  yield  a 

desirable  end  result. 
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Previous  sections  of  this  thesis  describe  how  the  role 
of  OPTEVFOR  has  become  more  encompassing  and  how  this  is 
causing  undesirable  duplication  among  test  agencies.   This 
appears  to  be  the  start  of  the  evolutionary  process  that 
is  predicted  herein. 

Since  OSD  and  DD(T&E)  place  primary  credibility  on  T&E 
results  obtained  from  OPTEVFOR,  Navy  Program  Managers  may 
soon  feel  compelled  to  task  OPTEVFOR  to  participate  in  all 
Naval  aircraft  T&E.   If  this  occurs,  when  a  Program  Manager 
has  a  system  change  that  needs  military  T&E,  he  will 
probably  request  that  only  OPTEVFOR  conduct  the  T&E.   The 
Program  Manager  could  have  both  a  DT&E  agency  and  OPTEVFOR 

conduct  the  T&E,  or  he  might  even  try  to  arrange  to  have 

tlJ 

a  joint  OPTEVFOR/DT&E  agency  team  conduct  the  evaluation. 

However,  both  these  alternatives  would  cost  more  and  take 
more  time,  since  two  T&E  agencies  vice  one  will  have  to 
analyze  the  data  and  report  their  results.   So  the  Program 
Manager  will  make  the  cost-effective  decision  and  have  only 
one  agency,  OPTEVFOR,  conduct  the  evaluation.   Since  the 
DT&E  agency  works  for  the  developing  agency,  the  Program 
Manager  should  have  little  trouble  in  squelching  the  dissen- 
sion that  comes  from  the  neglected  DT&E  agency. 

As  this  process  continues  the  DT&E  agencies  might  be 
forced  to  cut  back  personnel  due  to  decreased  work  loads, 
while  OPTEVFOR  increases  its  size.   OPTEVFOR  might  be  forced 
to  establish  its  own  test  pilot  school  to  teach  the  discipline 
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of  T&E.   Soon  OPTEVFOR  will  have  technical  capabilities 
that  make  it  the  only  one  capable  of  conducting  many  types 
of  T&E. 

This  evolution  would  continue  until  concern  arises  over 
the  highly  technical  T&E  that  is  being  conducted  by  OPTEVFOR. 
As  a  result  a  suggestion  might  appear  to  once  again  create  a 
completely  separate  organization  composed  of  military  person- 
nel with  good  operational  backgrounds.   These  personnel  would 
conduct  T&E  to  determine  if  a  weapon  system  is  suitable  for 
service  use  and  to  develop  the  optimum  employment  tactics. 
This  new  type  of  testing  might  be  called  Tactical  Test  and 
Evaluation  (TT&E). 

An  examination  of  the  current  T&E  environment  shows  that 

such  an  evolution  is  well  underway: 

OPTEVFOR  is  getting  more  and  more  projects. 

OPTEVFOR' s  Personnel  complement  is  increasing. 

More  technical  expertise  is  being  obtained  by  OPTEVFOR. 

OPTEVFOR  is  being  given  more  responsibilities  while  the 
DT&E  agencies  remain  fairly  stagnant. 

The  need  for  an  OT&E  School  is  becoming  more  apparent. 

The  need  for  a  larger  OPTEVFOR  civilian  cadre  is  becomingX 
more  apparent. 


/ 
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V.   PROPOSAL  FOR  A  TEST  AND  EVALUATION 
ORGANIZATIONAL  CHANGE 

Changes  are  constantly  being  made  in  order  to  improve 
Naval  aircraft  T&E,  but  the  thought  of  a  major  reorganization 
of  the  T&E  activities  has  always  been  met  with  a  high  degree 
of  apprehension.   However,  many  problems  facing  Naval  air- 
craft T&E  will  not  be  eliminated  unless  a  major  reorganiza- 
tion takes  place  in  the  T&E  community.   The  less  than 
optimal  utilization  of  the  Navy's  T&E  resources  has  been 
depicted,  and  many  of  the  causes  have  been  cited.   Since 
T&E  agencies  having  similar  responsibilities  report  to 
completely  separate  commands,  it  is  reasonable  to  assume 
that  inefficiencies  in  terms  of  duplication  will  be  diffi- 
cult to  reduce.   The  belief  that  "more  testing  is  better, 
so  let's  continue  to  do  more  and  not  worry  so  much  about 
the  duplication"  is  a  simplistic  and  costly  solution  to  the 
Navy's  T&E  problems,  but  one  that  has  gained  considerable 
support . 

A  proposed  approach  to  improving  the  Navy's  T&E  of  air- 
craft weapon  systems,  while  achieving  an  optimal  allocation 
of  the  Navy's  T&E  resources,  is  the  creation  of  a  Naval 
Test  and  Evaluation  Command  that  would  have  responsibility 


for  all  DT&E  and  OT&E  conducted  by  the  Navy  with  the  excep- 
tion of  the  necessary  T&E  conducted  by  Navy  Laboratories  in 
fulfilling  their  research  and  development  mission.   The 
Commander  of  the  Naval  T&E  Command  (NTEC)  should  be  a  Vice 
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Admiral  who  also  serves  as  a  Director,  Major  Staff  Office 
(DMSO)  within  OPNAV  with  the  title  of  Director,  Naval  Test 
and  Evaluation.   Figure  7  shows  this  proposed  OPNAV  struc- 
ture.  It  should  be  noted  that  the  dotted  line  relationship 
between  the  Chief  of  Naval  Material  and  the  Director,  Naval 
T&E  enables  the  Naval  Material  Command  to  assign  DT&E  and 
OT&E  projects  to  the  Naval  T&E  Command.   Projects  requiring 
Navy  T&E  from  NAVAIR  will  be  managed  by  the  Chief  of  Naval 
Air  T&E,  and  T&E  projects  from  NAVSHIPS  (Naval  Ships  Systems 
Command)  will  be  managed  by  the  Chief  of  Naval  Ship  T&E. 
Those  T&E  projects  coming  from  the  other  systems  commands 
of  NAVMAT  (Naval  Electronic  System  Command,  Naval  Ordnance 
Systems  Command,  or  Naval  Facilities  Engineering  Command) 
or  from  NAVMAT  designated  Program  Managers  will  be  managed 
by  either  the  Chief  of  Naval  Air  T&E  or  the  Chief  of  Naval 
Ship  T&E  depending  upon  the  nature  of  the  project. 

The  proposed  organizational  change  would  put  the  Direc- 
tor, Naval  Test  and  Evaluation  (DNT&E)  in  a  newly  created 
Director,  Major  Staff  Office  billet  (OP-09A)  reporting 
directly  to  OP-09,  the  Vice  Chief  of  Naval  Operations 
(VCNO).   For  compliance  with  DOD  Directive  5000.3,  the 
Director,  Naval  T&E,  would  report  the  results  of  his  Naval 
T&E  Command's  evaluations  directly  to  CNO,  the  immediate 
superior  of  the  VCNO.   As  a  DMSO  directly  under  OP-09,  the 
Director,  Naval  T&E  would  be  able  to  maintain  the  necessary 
degree  of  independence  from  the  developing  agency  that 


83 


CNO 


I 


OP-090 
Navy 
Program 
Planning 


VCNO(OP-09) 


OP-091 
Information 
Systems 
Division 


OP-092 

Fiscal 

Management 

Division 


OP-093 

OP-094 

OP-095 

Tactical 

Command 

Office  of 

and 

Support 

ASW  and 

Electromagnetic 

Programs 

Tactical 

Programs 

Electromagnetics 

OP-096 

Systems 

Analysis 

Division 


OP-097 
Ship 
Acquisition 

and 
Improvement 


OP-098 

Director 
RDT&E 


I 


OP-099 
Naval 
Education 
and  Training 


OP-09A 
Director 
Naval  Test 

and 
Evaluation 


Commander  Naval 
Air  T&E 


1 


Commander  Naval 
Ship  T8£ 


PROPOSED   OPMAV   STRUCTURE,    INCLUDING    A   DIRECTOR,    NAVAL   TEST 

AND   EVALUATION 

FIGURE    7 


84 


should  be  required  for  the  Commander  of  a  T&E  organization. 
He  would  never  have  to  function  both  as  the  weapon  system 
sponsor/proponent  and  as  the  T&E  spokesman  to  the  VCNO,  as 
is  now  the  case  for  OP-O98.   The  data  and  judgments  from 
each  DMSO  reporting  to  the  VCNO  would  be  evaluated  as  they 
are  today.   However  added  to  the  inputs  from  the  other 
Major  Staff  Offices  would  be  the  Navy's  unfiltered  input 
from  the  independent  T&E  command.   This  added  voice  to  the 
VCNO  should  facilitate  the  making  of  better  system 
acquisition  decisions  at  the  VCNO/CNO  level. 

The  creation  of  a  DMSO  for  Naval  T&E  would  not  separate 
T&E  from  RDT&E.   Development  test  and  evaluation  must  be 
an  integral  part  of  the  iterative  research  and  development 
(R&D)  process  of  design-test-evaluate-redesign  that  con- 
tinues until  technical  uncertainties  and  development  prob- 
lems are  resolved.   This  must,  therefore,  be  the  responsi- 
bility of  the  development  contractor  and  the  developing 
agency.   The  Director,  RDT&E  (OP-O98)  would  remain  the  CNO 
focal  point  for  this  activity  and  for  all  R&D  that  is  occur- 
ring within  the  Navy  or  under  Navy's  funding. 

However,  Navy  conducted  T&E  would  be  organizationally 
separated  from  this  process  and  would  no  longer  be  conducted 
by  agencies  working  either  for  the  program  sponsor  or  the 
developing  agency.   This  would  place  Navy  conducted  T&E 
in  an  ideal  location,  organizationally,  independent  from 
the  developer  and  fully  capable  of  reporting  unbiased  test 
results  to  the  highest  decision  making  level  within  the  Navy. 


85 


A.   DESCRIPTION  OP  PROPOSED  T&E  COMMAND 

The  proposed  Naval  T&E  Command  (Fig.  8)  would  have  two 
deputies  reporting  directly  to  the  Director,  Naval  Test 
and  Evaluation.   One  deputy,  the  Chief  of  Naval  Aircraft 
Test  and  Evaluation  would  be  responsible  for  all  Navy  T&E 
of  aircraft  and  the  associated  weapons  and  ancillary  equip- 
ment.  The  other  deputy,  the  Chief  of  Naval  Ship  Test  and 
Evaluation,  would  be  responsible  for  all  surface  and  sub- 
surface ship  T&E  and  the  associated  weapons  and  ancillary 
equipment.   No  discussion  is  contained  in  this  thesis  on 
the  organization  or  responsibilities  of  the  proposed  Chief 
of  Naval  Ship  Test  and  Evaluation.   It  is  recommended  that 
a  separate  study  be  conducted  in  this  area  to  evaluate  the 
feasibility  of  such  an  organization  and  to  determine  a 
suitable  composition  for  such  an  organization. 

The  staff  of  the  Chief  of  Naval  Aircraft  Test  and 
Evaluation  (CNAT&E)  would  be  composed  of  personnel  who  would 
exclusively  monitor  and  provide  guidance  for  Range  coor- 
dination, operational  T&E  by  the  VX  squadrons,  education 
of  project  personnel  by  either  the  Test  Pilot  School  (TPS) 
or  the  System  T&E  Training  Schools  (STETS)  to  be  created 
at  each  Test  Center,  and  the  typical  staff  functions  of 
every  major  staff.   The  Commanders  of  the  test  centers 
and  one  test  facility  would  report  to  the  CNAT&E. 

1.   Naval  Air  Test  Center,  Patuxent  River,  Maryland 
The  Naval  Air  Test  Center  at  Patuxent  River  would 
be  responsible  for  the  complete  T&E  of  the  aircraft  vehicle 
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(i.e.  less  the  weapon  system)  and  the  associated  auxiliary 
equipments.   NATC  would  also  become  the  air  ASW  test  center, 
and  would  in  addition  perform  T&E  on  most  support  aircraft. 
NATC  would  be  composed  of  four  test  divisions  and  two  train- 
ing divisions.   No  consideration  is  given  in  this  thesis 
to  an  instrumentation  or  technical  services  division  which 
would  probably  be  required  at  each  major  test  center.   The 
Aircraft  Test  Division  would  absorb  the  responsibilities 
of  the  current  Flight  Test  and  Service  Test  Divisions. 
Aircraft  Test  would  be  responsible  for  the  flying  qualities, 
performance,  and  carrier  suitability  testing  of  fixed  and 
rotary  wing  aircraft  (Plight  Test's  current  responsibilities) 
and  for  the  developmental  T&E  of  propulsion  and  aircraft 
systems,  excluding  electrical,  armament,  and  avionics 
(Service  Test's  current  responsibilities). 

The  Naval  Air  Test  Facility  at  Lakehurst,  New  Jersey 
would  retain  its  responsibility  for  the  T&E  of  catapults, 
arresting  gear,  and  other  aircraft  launch  and  recovery 
equipments.   Having  NATF  work  directly  for  NATC  would  more 

N 

easily  facilitate  better  utilization  of  the  catapult  and 
arresting  gear  facilities  that  are  located  at  both  Patuxent 
River  and  Lakehurst. 

The  Anti-Submarine  Warfare  Systems  Test  (ASWST)  Division 
would  replace  the  current  Weapons  System  Test  Division  at 
NATC.   ASWST  would  be  responsible  for  the  developmental 
T&E  of  the  electrical  armament,  and  avionics  systems  in  all 
rotary  wing  and  fixed  wing  ASW  aircraft  and  in  support 
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aircraft,  such  as  AEW  (airborne  early  warning)  aircraft  and 
cargo  aircraft.   VX-1  would  retain  its  present  role  of  OT&E 
of  ASW  aircraft  systems  but  would  also  become  responsible 
for  the  OT&E  of  certain  support  aircraft.   VX-1  would  main- 
tain its  operational  type  organization  and  its  close  rela- 
tionship with  the  operational  forces. 

The  U.S.  Naval  Test  Pilot  School  (TPS)  would  con- 
tinue to  report  to  the  Commander,  NATC,  and  its  functions 
would  remain  unchanged.   Approximately  half  of  its  Naval 
officer  graduates  would  become  project  officers  in  the 
expanded  Aircraft  Test  Division,  and  the  remainder  would 
be  assigned  to  project  officer  billets  primarily  at  the 
systems  test  divisions  at  the  three  test  centers.   While 
TPS  would  remain  a  relatively  long  duration  school  (one  year) 
and  would  provide  T&E  training  to  all  of  the  Aircraft  Test 
Division's  project  officers  and  civilian  engineers  it  would 
only  provide  training  to  a  relatively  small  number  of  the 
project  officers  and  civilian  engineers  throughout  the  rest 
of  the  Naval  Aircraft  T&E  Command.   Therefore,  schools  of 
relatively  short  duration,  called  System  Test  and  Evaluation 
Training  Schools  (STETS),  would  be  established  at  each  test 
center  to  teach  the  fundamentals  of  T&E  and  provide  a  basic 
foundation  in  the  technologies  utilized  in  the  necessary 
warfare  specialities;  i.e.,  ASW,  air-to-air,  attack,  ECM, 
etc.  (Appendix  F  presents  the  basic  objectives  of  these 
schools).   A  STETS  would  be  established  at  the  Naval  Air 
Test  Center  to  teach  the  fundamentals  of  ASW  and  of  any 
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required  support  warfare  to  the  project  officers  and  civilian 
engineers/analysts  reporting  to  VX-1  and  to  the  ASW  Systems 
Test  Division.   Officers  that  just  graduated  from  TPS  might 
require  a  brief  supplementary  course  from  the  STETS  prior  to 
commencing  work  in  the  systems  test  division,  and  those 
officers  coming  back  for  second  tours  of  duty  in  T&E  might 
require  a  special  refresher  STETS  course. 

2.   Naval  Fighter  Test  Center,  Pt .  Mugu,  California 

The  Naval  Missile  Center  Complex  at  Point  Mugu  would 
become  the  Naval  Fighter  Test  Center  and  be  responsible  for 
the  T&E  of  all  air  to  air  weapons  systems  and  ancillary  and 
support  equipment.   The  Naval  Fighter  Test  Center  would  be 
composed  of  basically  four  divisions:   a  Fighter  Systems 
Test  (FST)  Division,  VX-4,  Fighter  System  T&E  Training 
School  (FSTETS),  and  PMR  (Pacific  Missile  Range)  Division. 

FST  would  be  responsible  for  the  developmental  T&E 
of  fighter  aircraft  weapons,  avionics,  and  electrical 
systems,  and  VX-1!  would  be  responsible  for  the  OT&E  of  the 
complete  fighter  aircraft  system  and/or  air  to  air  weapon 
system.   The  Fighter  STETS  would  provide  a  relatively  short 
course  in  T&E  of  fighter  weapon  systems  to  project  officers 
and  engineers/analysts  reporting  to  FST  and  VX-4.   It  would 
function  similar  to  the  STETS  at  the  Naval  Air  Test  Center. 

The  PMR  division  would  be  responsible  for  the  opera- 
tion and  management  of  the  Pacific  Missile  Range. 
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3.   Naval  Attack  System  Test  Center,  China  Lake, 
California 

The  China  Lake  complex  (excluding  the  Naval  Weapons 
Center  )  would  become  known  as  the  Naval  Attack  Systems 
Test  Center  (NASTC).   It  would  be  responsible  for  the  T&E 
of  all  attack  aircraft  weapon  systems,  and  all  ECM  systems 
and  reconnaissance  systems  (excluding  ASW) .   The  center's 
responsibilities  would  include  DT&E  of  the  aircraft  armament, 
avionics  and  electrical  systems  of  all  attack,  ECM,  and 
reconnaissance  aircraft,  the  OT&E  of  these  aircraft,  and  the 
complete  Navy  T&E  of  all  air  to  surface  weapons.   The  NASTC 
would  be  organized  similar  to  the  NFSTC  at  Pt .  Mugu. 

The  four  basic  divisions  of  the  NASTC  would  be 
Attack  Systems  Test  (AST),  VX-5,  Attack_STETS,  and  Range 
Division.   The  AST  would  be  responsible  for  the  developmental 
T&E  of  all  attack  aircraft  weapon  systems  and  weapons,  and 
VX-5  would  be  responsible  for  the  OT&E  of  the  complete  attack 
aircraft  and/or  air  to  surface  weapon.   The  Attack  STETS 
would  provide  the  necessary  attack,  ECM,  and  reconnaissance 
systems  T&E  training  for  VX-5  and  AST  project  officers  and 
engineers/analysts.   The  Range  division  would  be  responsible 
for  the  operation  and  management  of  the  various  China  Lake 
ranges . 


The  Naval  Weapons  Center  (NWC)  which  is  a  Navy  research 
and  development  laboratory  located  at  China  Lake  would  • 
continue  to  report  to  the  Chief  of  Naval  Research  and  be  the 
principal  RDT&E  Center  for  air  warfare  and  missile  systems. 
NWC  would  be  able  to  task  the  NASTC  to  conduct  DT&E  and 
OT&E. 


91 


h.      Naval  Weapons  Evaluation  Facility  (NWEF) , 
Albuquerque,  New  Mexico 

HWEP  would  retain  its  current  function  of  T&E  of 

nuclear  weapons  and  the  associated  systems  within  the 

compatible  delivery  aircraft.   NWEF  would  be  subdivided 

into  a  technical  division,  Nuclear  Weapons  Systems  Test, 

and  an  OT&E  division,  and  a  VX  squadron.   As  a  result  of  the 

relatively  small  size  of  NWEF  a  formal  STETS  would  not  be 

established.   Instead  its  project  officers  and  engineers/ 

analysts  would  receive  T&E  training  at  the  Attack  STETS  at 

China  Lake  and  the  necessary  nuclear  technical  training 

through  an  informal  NWEF  training  course. 

B.   DT&E  PERSPECTIVE 

Discussion  of  the  proposed  reorganization  with  personnel 
in  the  Naval  Air  Systems  Command  was  usually  met  with  dis- 
sension.  Their  belief  that  the  developing  agency  requires 
absolute  control  over  all  DT&E  is  unfounded.   Most  aircraft 
DT&E  is  conducted  by  the  contractor,  and  it  is  only  that 
portion  of  DT&E  conducted  by  the  Navy  that  is  in  dispute. 
Under  the  concept  proposed  in  this  thesis,  the  developing 
agency  could  task  the  Naval  T&E  Command  to  perform  whatever 
type  T&E  is  required.   The  funding  for  the  T&E  would  be 


provided  by  the  developing  agency,  and  the  Naval  Air  T&E 
Command  would  report  its  T&E  results  to  the  developer, 
exactly  as  is  done  today. 

From  the  developer's  perspective  little  change  will 
have  actually  occurred  other  than  the  DT&E  agencies  would 
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no  longer  work  directly  for  the  developer.   The  Program 
Manager,  the  key  man  in  managing  the  development  and  produc- 
tion of  a  weapon  system,  would  operate  in  the  NAVMAT  Matrix 
organization  in  the  same  way  as  he  does  today.   The  PM 
would  still  task  the  various  functional  NAVAIR  Divisions, 
other  System  Command  Divisions,  and  civilian  contractors 
to  perform  essential  functions  for  his  program.   However, 
instead  of  tasking  a  NAVAIR  agency  to  perform  DT&E  and 
OPTEVFOR  to  perform  OT&E,  he  would  task  the  Naval  T&E 
Command  to  perform  all  of  his  Navy  T&E. 

The  developing  agency  could  task  the  Naval  Air  T&E 
Command  to  conduct  only  test  and  analysis  rather  than  test 
and  evaluation.   This  would  permit  the  developing  agency 
to  merely  task  the  T&E  Command  to  obtain  specific  test  data, 
but  not  to  evaluate  the  significance  of  the  data.   This 
type  of  request  from  the  developing  agency  might  occur  if 
the  fleet  were  to  lose  one  or  more  aircraft  for  an  unexplained 
reason  (e.g.,  the  wing  falling  off  in  flight,  but  all  inci- 
dents occurring  over  water  and,  therefore,  no  evidence  exists 
which  would  point  to  the  cause).   In  such  a  case  NAVAIR 
would  probably  task  the  Naval  Air  T&E  Command  to  perform 
numerous  structural  tests  and  analysis  and  merely  send  the 
various  load  data  to  NAVAIR  for  its  evaluation,  which  would 
probably  be  done  jointly  with  the  particular  airframe 
manufacturer. 

At  present,  much  of  the  development  testing  is  planned 
by  the  Naval  Air  System  Command,  and  very  detailed 
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descriptions  of  the  tests  and  test  conditions  are  given  to 
the  DT&E  agencies  in  the  AIRTASKS,  the  official  NAVAIR 
document  which  assigns  work  and  provides  funding  to  the 
DT&E  agencies  under  NAVAIR' s  control.   This  same  type 
procedure  should  work  when  NAVAIR  tasks  the  T&E  Command  to 
perform  development  tests. 

With  Congressional  pressure  to  decrease  the  military's 
Washington  D.C.  personnel,  NAVAIR  is  facing  severe  cutbacks 
which  could  have  an  adverse  effect  on  maintaining  T&E 
expertise.   The  creation  of  a  T&E  Command  would  provide 
a  simple  means  which  would  permit  NAVAIR  to  decentralize 
this  expertise.   Many  NAVAIR  engineers  could  be  transferred 
to  the  T&E  Command  which  would  accomplish  two  objectives: 

(1)  It  would  decrease  the  number  of  Navy  personnel 
in  Washington  D.C. 

(2)  It  would  place  the  majority  of  the  Navy's  aircraft 
T&E  expertise  in  the  test  agency. 

In  the  current  system  much  of  the  Navy's  T&E  expertise 
exists  in  NAVAIR,  and  consequently  NAVAIR  does  much  of 
the  test  evaluation,  and  the  test  centers  do  too  little 
actual  test  evaluation. 

Since  the  development  agency  is  responsible  for  develop- 
ing a  system  that  works,  meets  specifications,  and  is  opera- 
tionally suitable,  the  timely  and  effective  conduct  of  OT&E 
is  as  much  a  responsibility  of  the  Program  Manager  as  is 
the  timely  and  effective  conduct  of  DT&E.   Under  the  proposed 
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T&E  organizational  concept,  the  development  agency  would 
be  able  to  task  the  T&E  command  to  perform  OT&E  in  the  same 
way  as  it  would  task  the  T&E  Command  to  perform  DT&E. 

At  present  the  Program  Manager  is  essentially  the  NAVAIR 
decision  point  on  where  to  conduct  various  tests  for  pro- 
grams that  are  not  tracked  by  DSARC.   It  is  unlikely  that 
each  Program  Office  has  sufficient  T&E  expertise  to  always 
make  test  location  decisions  which  contribute  to  optimizing 
the  utilization  of  the  Navy's  T&E  resources.   Program 
Manager's  T&E  decisions  on  where  to  conduct  various  tests 
are  influenced  unduly  by  the  program's  dollar  and  time  con- 
straints with  too  little  regard  for  the  expertise  of  the 
test  agency. 

In  the  current  T&E  structure,  the  developing  agency's 
Program  Manager  has  three  alternatives  when  he  requires 
that  OT&E  be  performed: 

(1)  Task  a  DT&E  agency  to  perform  the  tests  and  call 
the  tests  "testing  for  service  suitability"  vice 
OT&E 

(2)  Generate  a  request  to  CNO  to  have  a  Development 
Assist1  project  assigned  to  OPTEVFOR  to  conduct  the 
desired  OT&E 

(3)  Request  that  an  operational  unit  (i.e.  an  aircraft 
squadron,  ship,  etc.)  evaluate  the  item  under 
operational  conditions. 


Development  Assists  are  projects  which  are  assigned 
by  CNO  at  the  request  of  the  Developing  Agency  to  OPTEVFOR 
[Ref.  26]. 
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In  the  proposed  T&E  organization  the  Program  Manager 
would  no  longer  have  the  flexibility  in  deciding  which 
organization  should  do  Navy  T&E.   The  Naval  T&E  Command  would 
make  that  decision  and  be  expected  to  optimize  the  utiliza- 
tion of  the  Navy's  T&E  resources  while  obtaining  credible 
T&E  results. 

One  criticism  of  the  proposed  change  has  been  the  size 
of  the  testing  command  that  would  be  created  and  the  many 
problems  that  are  typical  of  large  organizations.   Care  must 
be  exercised  to  minimize  creation  of  "paper  work"  stumbling 
blocks  which  are  so  easy  to  generate  in  large  organizations. 
Prior  to  the  establishment  of  a  T&E  Command,  a  study  should 
be  conducted  to  determine  the  best  way  to  organize  such  a 
command  while  maintaining  testing  efficiency,  especially  in 
regard  to  timely  completion  of  assigned  T&E  including 
publishing  the  reports. 

One  of  the  major  problems  facing  Program  Managers  is 
obtaining  T&E  results  within  the  time  constraints  specified 
in  the  tasking  document  (e.g.,  AIRTASK).   As  discussed  in 
Section  IV. B. 6,  the  current  T&E  organization  is  frequently 
guilty  of  late  completion  of  T&E  reports.   A  fear  voiced  by 
Program  Managers  regarding  the  creation  of  a  T&E  command  is 
the  potential  problem  of  its  unwillingness  to  perform  T&E 
in  accordance  with  the  Program  Manager's  time  requirements 
with  the  Program  Manager  having  no  capability  to  order  the 
T&E  agency  to  perform  as  directed.   Theoretically,  this  so 
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called  "ordering"  can  be  done  now,  but  in  practice  this 
requires  a  dialogue  between  the  Commander,  Naval  Air 
Systems  Command  and  the  Commander  of  the  test  agency.   The 
decision  as  to  whether  to  comply  with  the  Program  Manager's 
urgent  T&E  requirement  will  then  consider  the  overall  agency 
T&E  workload  and  NAVAIR's  overall  requirements.   In  the 
proposed  structure  the  Program  Manager  would  hve  to  resort 
to  different  means  to  induce  this  same  type  of  high  level 
dialogue  to  determine  the  relative  priority  of  his  project. 
However,  the  central  control  of  T&E  should  reduce  the 
problem  of  determining  relative  project  priorities,  since 
random  inputs  from  CNO  to  accomplish  "his"  special  project 
would  be  much  less  likely,  and  these  projects  often  cause 
arbitrary  priorities. 

An  examination  today  of  the  responsiveness  of  OPTEVFOR 
and  the  timeliness  of  its  completing  T&E  and  publishing 
reports  should  dispel  any  doubts  that  an  "independent"  T&E 
agency  would  respond  as  rapidly  for  the  developer  as  a  T&E 
agency  that  works  for  the  developer.   OPTEVEOR  does  respond 
as  rapidly  and  publishes  its  reports  probably  more  promptly 
than  the  NAVAIR  controlled  DT&E  agencies. 

C.   OT&E  PERSPECTIVE 

One  of  the  big  concerns  of  OPTEVFOR  is  the  need  to 
maintain  a  close  identity  with  the  operational  community. 
There  is  a  concern  that  if  OT&E  becomes  too  technical,  it 
can  no  longer  be  truly  operational.   By  adopting  the 
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proposed  T&E  organizational  change,  the  VX  squadrons  would 
be  better  able  to  maintain  their  valuable  operational 
perspective,  since  they  would  not  be  compelled  to  develop 
an  imbalance  of  technical  expertise  over  operational 
expertise  as  appears  to  be  happening  today.   Since  the  VX 
squadrons  are  being  tasked  to  conduct  some  T&E  solely  on 
the  basis  of  their  "independence,"  rather  than  their 
expertise,  OPTEVFOR  is  compelled  to  become  more  technically 
oriented  in  order  to  perform  capably  the  increased  technical 
T&E  requirements.   The  danger  of  having  this  continue  to  the 
point  that  OPTEVFOR  develops  an  overbalance  on  the  technical 
aspects  of  T&E  should  be  obvious. 

The  concept  of  having  the  VX  squadrons  operate  very  much 
like  an  operational  squadron  in  order  to  maintain  a  better 
operational  perspective  would  be  retained.   The  VX  squadrons 
would  work  closely  with  the  co-located  systems  test  division 
(ASW,  fighter,  or  attack)  throughout  an  aircraft's  develop- 
ment.  T&E  reports  from  each  of  the  test  centers  could 
contain  solely  the  results  of  DT&E  or  OT&E  or  the  reports 
could  contain  the  results  of  combined  testing  (DT&E  and 
OT&E).   The  combined  T&E  reports  would  have  a  technical 
section  (for  DT&E)  and  an  operational  section  (for  OT&E), 
and  differing  opinions  in  each  section  could  occasionally 
be  expected.   However,  the  CNAT&E  staff  would  comment  on 
reports  having  significantly  different  findings  in  the 
technical  and  operational  sections  in  order  to  aid  DD(T&E), 
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DSARC,  Program  Manager,  or  other  appropriate  decision 
makers  in  reconciling  the  different  opinions. 

In  the  past  the  biggest  contribution  made  by  the  VX 
squadrons  was  in  the  area  of  tactics  development.   This 
very  important  area  cannot  be  overlooked,  and  the  proposed 
organization  would  ensure  that  tactics  development  remains 
a  major  function  of  each  VX  squadron.   By  having  the  VX 
squadrons  involved  earlier  in  weapon  system  development  they 
would  be  better  prepared  to  develop  sound  tactics  for  each 
weapon  system  much  sooner  than  in  the  past.   (Previously 
it  often  took  more  than  a  year  of  fleet  operation  with  a 
weapon  system  before  a  VX  squadron  released  its  report  on 
optimum  use  procedures  and  employment  tactics). 

When  personnel  are  involved  in  finding  deficiencies  and 
recommending  changes  to  a  weapon  system  throughout  its 
development  cycle,  they  very  often  develop  strong  feelings 
against  certain  features  which  they  reported  as  being  defi- 
cient, and  which  are  often  not  corrected.   It  is  important 
to  have  the  people  who  are  developing  tactics  for  a  weapon 
system  fully  believe  in  and  be  strong  proponents  of  the 
potential  of  that  weapon  system.   This  is  a  potential  prob- 
lem area  that  exists  in  the  current  T&E  process  where  the 
OT&E  project  officers  are  being  put  more  into  the  role  of 
developmental  testing,  in  that  they  perform  tests  during 
the  various  iterations  of  design,  and  the  specific  defi- 
ciencies they  cite  assist  in  the  evolution  of  the  design. 
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This  type  of  involvement  has  often  caused  a  DT&E  project 
officer  to  develop  strong  dissatisfactions  toward  the  weapon 
system  he  tested. 

The  proposed  T&E  organization  would  not  put  the  OT&E 
project  officer  into  this  design  iteration  testing  process, 
but  would  allow  him  to  view  more  readily  the  development 
of  the  entire  system  and  not  each  subsystem  design  iteration. 
This  will  enable  the  OT&E  project  officer  to  maintain  a 
better  operational  perspective  of  the  entire  system  while 
becoming  involved  with  the  system  early  enough  to  develop 
tactics  on  a  timely  basis. 

Perhaps  the  most  overlooked,  but  often  most  valuable, 
portion  of  OT&E  is  the  de  facto  OT&E  that  is  conducted 
during  the  early  deployments  of  a  weapon  system.   As  hard 
as  the  Navy  tries  to  move  OT&E  earlier  into  the  development 
cycle,  it  can  never  eliminate  the  value  of  the  actual  de- 
ployment of  the  weapon  system  in  the  operational  environ- 
ment.  At  present  the  operational  command  using  the  system 
is  often  confused  as  to  whom  he  should  report  system  prob- 
lems.  NAVAIR  is  naturally  cognizant  for  aircraft  systems 
but  various  T&E  agencies  often  would  be  able  to  make  a  rapid 
and  valuable  contribution  if  they  were  informed  of  the 
problem.   The  user  consequently  informs  NAVAIR  but  is  usually 
uncertain  as  to  whether  to  also  contact  OPTEVFOR,  the  appro- 
priate VX  squadron,  NATC,  NWC ,  NMC ,  or  NWEF .   The  proposed 
organization  would  facilitate  better  retrieval  of  operational 
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data  from  the  fleet  by  eliminating  the  operational  command's 
dilemma  in  trying  to  determine  which  T&E  agency  is  cognizant 
of  each  problem  that  arises. 

D.   T&E  ORGANIZATIONAL  CHANGE  WILL  CORRECT  CURRENT  T&E 
DEFICIENCIES 

Many  of  the  T&E  deficiencies  cited  in  Section  IV  of 
this  thesis  would  be  corrected  by  the  adoption  of  the  pro- 
posed T&E  organizational  change  as  follows: 

1.   Organizational  Deficiencies 

The  creation  of  a  DMSO  for  Naval  T&E  would  ensure 
that  proper  high  level  attention  is  given  to  independent 
T&E  results.   Since  the  Director,  Naval  Test  and  Evaluation 
would  not  be  assigned  as  a  program  sponsor,  he  would  not 
be  forced  to  change  hats  between  the  user  and  the  producer 
(advocacy)  roles  as  now  occurs  in  OPNAV  with  OP-098. 

Having  the  Director,  Naval  T&E  report  to  OP-09 
would  preserve  the  test  organization  independence  that  is 
required  by  directive.   This  independence  also  appears 
logical  and  makes  intuitive  sense  for  a  test  organization. 
In  many  aircraft  companies  the  Chief  Test  Pilot  reports 
to  the  president  as  often  does  the  Product  Assurance  Divi- 
sion.   The  entire  T&E  command  can  be  thought  of  as  a 


Product  Assurance  is  the  term  now  given  to  many  com- 
panies' previous  Quality  Assurance  Divisions,  since  Product 
Assurance  is  meant  to  be  more  encompassing.   Product  assur- 
ance includes  the  disciplines  of  design  assurance,  maintain- 
ability, reliability,  test  and  evaluation,  and  safety. 


101 


Product  Assurance  group  that  ensures  weapon  systems  meet 
specifications  and  are  operationally  suitable.   Reporting 
of  such  an  organization  to  a  high  level  ensures  unfiltered 
test  results. 

Duplication  of  T&E  responsibilities  would  be  more 
easily  reduced  if  all  Navy  T&E  is  controlled  within  one 
Command.   The  T&E  Command  would  install  among  the  various 
DT&E  agencies  central  line  management  control,  which 
doesn't  exist  today  in  the  NAVAIR  T&E  coordinator. 

Since  only  one  T&E  Command  would  exist  in  the  Navy, 

the  flexibility  of  where  to  assign  T&E  projects  would  no 

longer  rest  with  the  Program  Manager,  but  with  the  T&E 

Command.   This_would  help  ensure  that  adequate  T&E  is 

-"  6 

performed  on  non-major  procurements. 

The  creation  of  a  T&E  Command  which  is  entirely 
independent  from  the  developing  agencies  would  eliminate 
the  dilemma  facing  DD(T&E)  today.   He  would  be  able  to  place 
equal  credibility  on  all  test  results  from  the  T&E  Command, 
and  would  not  have  to  request  an  operational  test  agency 
to  repeat  a  technical  test  merely  to  ensure  that  the  re- 
quired degree  of  independence  is  maintained  in  the  test 
results . 

Placing  all  T&E  agencies  under  one  command  would 
facilitate  the  creation  of  a  uniform  system  of  funding 
throughout  the  testing  community.   It  would  also  simplify 
the  task  of  identifying  all  the  funding  required  for  T&E. 
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With  all  the  test  agencies  under  one  command,  all 
ranges  would  also  be  controlled  by  the  same  command.   This 
central  control  of  ranges  would  facilitate  easier  identifi- 
cation of  effective  and  efficient  range  use  and  funding 
requirements . 

2.   Testing  Deficiencies 

Much  of  the  overlap  between  OT&E  and  DT&E  within 
the  Navy  has  been  caused  by  the  need  for  independence  being 
identified  with  only  OT&E.   By  having  all  T&E  independent 
of  the  developing  agency  the  different  perspectives  of  the 
OT&E  and  DT&E  roles  can  be  more  easily  retained. 

Having  T&E  Command  visibility  at  the  VCNO  level 
would  ensure  that  test  planning  is  given  more  attention 
and  T&E  experts  become  involved  in  this  planning.   This 
high  level  visibility  should  ensure  the  creation  of  better 
operational  test  objectives. 

The  problem  of  inadequate  technical  expertise  in 
the  OT&E  test  agency,  OPTEVPOR,  would  be  somewhat  reduced 
as  a  result  of  the  elimination  of  the  need  for  the  OT&E 
agency  to  expand  its  testing  expertise  into  more  developmen- 
tal type  testing.   The  creating  of  STETS  (System  T&E  Train- 
ing Schools)  at  each  test  center  would  provide  a  place  to 
give  all  project  officers  and  civilian  engineers/analysts 
a  basic  background  in  T&E,  and  should  therefore  improve 
the  entire  OT&E  and  DT&E  process. 

Late  test  results  is  a  problem  that  would  not  be 
corrected  by  the  proposed  organization  alone.   Since  each 
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test  center  would  be  organized  similarly,  each  test  center 
would  be  motivated  to  develop  the  most  efficient  T&E 
report  review  process.   However,  this  area  should  be  given 
considerable  attention  when  creating  the  Naval  T&E  Command. 

With  a  T&E  Command  that  is  independent  from  the 
developer  and  that  is  capable  of  ensuring  specification 
compliance,  there  would  no  longer  be  a  need  for  BIS.   It 
is,  therefore,  recommended  that  BIS  be  disestablished  if 
this  thesis  proposal  is  adopted. 

By  having  the  VX  squadrons  within  the  T&E  Command, 
the  need  for  analysts  throughout  the  T&E  Command  would  become 
obvious.   As  the  need  for  more  analysts  within  the  T&E 
community  is  recognized  a  better  career  pattern  for  analysts 
within  Navy  T&E  should  evolve.   This  would  increase  the 
ability  of  Navy  T&E  to  attract  and  retain  high  caliber 
analysts. 

By  having  one  T&E  Command  it  will  be  easier  to 
ensure  continual  monitoring  of  development  programs.   The 
infrequent  snap  shot  looks  at  a  program  by  each  of  the  Navy 
agencies  which  often  occurs  today  was  shown  to  be  an  unsat- 
isfactory method  for  assessing  system  potential.   The  pro- 
cedure of  assigning  a  resident  T&E  team  composed  of  both 
DT&E  and  OT&E  personnel  during  a  systems'  early  development 
Is  being  utilized  by  the  Air  Force  on  the  B-l  program 
[Ref.  27]  and  was  utilized  on  the  F-15  program.   This 
procedure  has  considerable  merit  and  will  be  relatively 
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easy  to  implement  if  all  Navy  T&E  personnel  were  assigned 
to  one  Naval  T&E  Command. 

The  creation  of  a  Naval  T&E  Command  will  give 
increased  stature  to  all  T&E  within  the  Navy.   This  should 
make  T&E  duty  for  Naval  Officers  more  highly  desirable  for 
second  and  third  tours  of  duty  than  it  is  today.   Effective 
military  T&E  can  only  be  planned  and  accomplished  by  per- 
sonnel who  have  operational  and  technical  backgrounds  and 
testing  experience.   Too  little  emphasis  in  the  past  has 
been  placed  upon  encouraging  good  officers  to  become 
specialized  in  the  T&E  field  and  to  develop  that  necessary 
testing  experience.   As  a  result,  many  middle  and  top  mana- 
gers within  the  T&E  agencies  are  not  of  the  caliber  necessary 
to  manage  the  type  of  effective  and  efficient  T&E  that  the 
Navy's  limited  resources  demand.   The  creation  of  a  Naval 
T&E  Command  will  hopefully  do  much  to  eliminate  this 
deficiency. 


* 
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VI.   CONCLUSIONS  AND  RECOMMENDATIONS 

A.   CONCLUSIONS 

1.  The  valuable  internal  "user-producer  dialogue"  is 
lost  during  the  latter  stages  of  a  weapon  system's  develop- 
ment, since  the  Navy's  independent  T&E  organization,  OPTEVFOR, 
reports  to  OP-O98  who  is  compelled  to  act  as  a  system  propo- 
nent and  must  often  conceal  weapon  system  development 
problems. 

2.  Neither  the  development  T&E  agencies  nor  OPTEVFOR 
is  independent  from  the  influence  of  the  developing  agency. 

3.  An  excessive  amount  of  duplication  exists  in  the 
testing  conducted  by  the  major  participants  in  the  Navy 
aircraft  weapon  system  T&E  process. 

k.  Inadequate  central  management  exists  among  the  NAVAIR 
T&E  activities,  consequently  costly  duplication  of  facilities 
and  expertise  exists  among  the  NAVAIR  T&E  activities. 

5.  OPTEVFOR  is  being  tasked  to  perform  T&E  often  on  the 
basis  of  its  independence  rather  than  on  its  testing 
capability. 

6.  Adequate  checks  do  not  exist  in  the  Navy's  acquisi- 
tion process  for  non-major  systems  to  ensure  that  adequate 
T&E  is  conducted  and  that  the  major  deficiencies  are 
corrected  prior  to  procuring  and  deploying  the  system. 

7.  The  costs  for  conducting  T&E  cannot  be  totally 
identified,  and  numerous  methods  are  utilized  by  the  T&E 
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agencies  to  calculate  the  costs  of  their  performing  T&E. 
This  lack  of  standard  cost  accounting  makes  T&E  cost- 
effectiveness  decisions  nearly  impossible  to  perform. 

8.  Central  aircraft  T&E  range  management  does  not 
exist,  consequently  it  is  a  difficult  task  to  justify  the 
various  ranges  and  make  rational  decisions  when  forced  to 
reduce  range  assets. 

9.  A  clear  distinction  between  the  responsibilities 
of  DT&E  and  OT&E  does  not  exist,  consequently  testing 
duplication  is  increasing. 

10.   Test  agencies  do  not  participate  adequately  in 
early  T&E  planning. 

11. .  Adequate  criteria  against  which  OT&E  can  be 
conducted  often  does  not  exist. 

12.  Inadequate  technical  expertise  exists  in  OPTEVFOR 
to  effectively  perform  all  of  its  responsibilities.   This 
problem  has  been  magnified  by  the  increased  T&E  responsibil- 
ity given  to  OPTEVFOR. 

13.  A  definite  requirement  exists  for  a  technical  course 
of  instruction  in  the  discipline  of  Weapon  System  T&E. 

14.  Program  Managers  frequently  do  not  receive  test 
results  from  the  various  test  agencies  soon  enough  to  assist 
in  making  vital  program  decisions. 

15.  OPNAVINST  3960.8  shows  BIS  commencing  along  with 
OPEVAL  prior  to  the  DSARC  III  major  production  decision. 
This  sequencing  is  illogical  if  BIS  and  OPEVAL  are  to  retain 
their  current  roles. 
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16.  A  shortage  of  statistical  analysts  exists  throughout 
the  T&E  community  (DT&E  and  OT&E). 

17.  Test  and  evaluation  is  often  treated  as  a  process 
where  snapshot  looks  at  a  system  can  give  valid  conclusions 
as  to  the  system's  development  progress.  This  is  false, and 
T&E  must  be  treated  as  a  continuing  process.. 

18.  Many  good  characteristics  continue  to  exist  in  the 
Navy's  T&E  process. 

a)  Navy  test  pilots  and  NFO's  typically  have  and 
maintain  excellent  operational  backgrounds 

b)  Testing  is  conducted  by  relatively  small 
organizations 

c)  Competition  among  test  agencies  stimulates  good 
performance 

d)  Decentralized  test  management  is  practiced 

e)  T&E  attracts  top  caliber  military  personnel 

B .   RECOMMENDATIONS 

1.  The  Navy  should  create  a  Naval  T&E  Command  that 
would  have  responsibility  for  all  Navy  conducted  T&E  and 
OT&E  with  the  exception  of  the  necessary  T&E  conducted  by 
Navy  laboratories  in  fulfilling  their  R&D  missions. 

2.  The  Commander  of  the  Naval  T&E  Command  should  also 
serve  as  a  Director,  Major  Staff  Office  reporting  to  OP-09, 
the  VCNO. 


. 


3.   The  Naval  Air  T&E  Command  should  be  organized  as 
proposed  in  this  thesis. 

k.      A  study  should  be  conducted  to  determine  the 
appropriate  organizational  structure  and  functions  of  a 
Naval  Ship  T&E  Command. 
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5.   A  System  T&E  Training  School  should  be  established 
at  each  of  the  major  test  centers. 
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APPENDIX  A 
ORGANIZATIONS  WITH  WHICH  INTERVIEWS  WERE  CONDUCTED 

1.   Office  of  the  Secretary  of  Defense,  Washington,  D.C, 
DDR&E  (T&E) 


2.  Office  of  the  Secretary  of  the  Navy,  Washington,  D.C. 

ASN  (R&D) 

3.  Office  of  the  Chief  of  Naval  Operations,  Washington,  D.C 

OP-983 
OP-506 

OP-508 

4.  Naval  Air  Systems  Command,  Washington,  D.C. 

AIR  05 

AIR  510 

PMA-241 

PMA-2^4 

NAVAIR  T&E  Coordinator,  Patuxent  River,  Md. 

5.  Senate  Armed  Services  Committee 

RDT&E  Subcommittee 

6.  Naval  Air  Test  Center,  Patuxent  River,  Md. 

7.  Sub  Board  of  Inspection  and  Survey,  Patuxent  River,  Md. 

8.  Operational  Test  and  Evaluation  Force 

Headquarters,  Norfolk,  Va. 

Air  Development  Squadrons  One,  Four  and  Five 

Deputy  COMOPTEVFOR,  North  Island,  Ca. 
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APPENDIX  B 
DEFINITIONS  OF  DEVELOPMENT  AND  OPERATIONAL  TEST  AND  EVALUATION 

Development  Test  and  Evaluation  (DT&E) 

DT&E  is  that  T&E  conducted  under  the  sponsorship  of  the 

developing  agency  to  facilitate  in  the  evolution  of  the 

system.   Its  specific  functions  are  to: 

"(1)  demonstrate  that  the  engineering  design  and  development 
process  is  complete. 

(2)  demonstrate  that  the  design  risks  have  been  minimized. 

(3)  demonstrate  that  the  system  will  meet  specifications. 

(4)  estimate  the  system's  military  utility  when  introduced." 
[Ref .  6  ] 

The  types  of  tests  included  in  DT&E  are:   engineering 
tests,  contractor/laboratory  demonstrations,  Navy  Technical 
Evaluations  (NTE),  Navy  Preliminary  Evaluations  (NPE),  and 
deficiency  correction  tests.   Advanced  development,  engineer- 
ing and  production  prototype  models  are  used  for  DT&E. 


Operational  Test  and  Evaluation  (OT&E) 

"OT&E  is  that  test  and  evaluation  conducted  to  estimate 
the  prospective  system's  military  utility,  operational 
effectiveness,  and  operational  suitability  (including 
compatibility,  interoperability,  reliability,  maintain- 
ability, and  logistic  and  training  requirements),  and 
need  for  any  modifications.   In  addition,  OT&E  provides 
information  on  organization,  personnel  requirements, 
doctrine,  and  tactics  .  .  .  OT&E  will  be  accomplished 
by  operational  and  support  personnel  of  the  type  and 
qualifications  of  those  expected  to  use  and  maintain 
the  system  when  deployed,  and  will  be  conducted  in  as 
realistic  an  operational  environment  as  possible." 
[Ref.  6] 
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OT&E  is  divided  into  two  categories,  initial  operational 
test  and  evaluation  (IOT&E)  and  follow-on  operational  test 
and  evaluation  (FOT&E).   IOT&E  is  that  OT&E  conducted  prior 
to  the  first  major  production  decision  (DSARC  III).   FOT&E 
is  the  continuing  OT&E  of  a  weapon  system  after  the  produc- 
tion decision  has  been  made. 
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APPENDIX  C 

INSTRUCTIONS/DIRECTIVES  PERTAINING  TO  THE  T&E  OF  NAVAL 

AIRCRAFT  WEAPON  SYSTEMS 


DOD  Directive  5000.1,  "Acquisition  of  Major  Defense  Systems", 
13  July  1971 

DOD  Directive  5000.3,  "Test  and  Evaluation",  19  July  1973 

SECNAV  Instruction  5000.1,  "System  Acquisition  in  the 
Department  of  the  Navy",  13  March  1972 

OPNAV  Instruction  3930. 8B  "Assignment  and  Prosecution  of 
Test  and  Evaluation  Projects",  5  April  1973 

OPNAV  Instruction  3960.8  "Test  and  Evaluation  of  Navy 
Systems  and  Equipments",  22  January  1973 

OPNAV  Instruction  5*120.70,  "Mission,  Organization  and 

Function  of  the  Board  of  Inspection  and  Survey", 
2  April  1971 

OPNAV  Instruction  5440. ^7D,  "Mission  and  Functions  of 

Operational  Test  and  Evaluation  Force  (OPTEVFOR)", 
6  March  1973 

NAVAIR  Instruction  5^00. 27B,  "Assignment  of  Coordination 
Responsibilities  for  the  Naval  Air  Systems 
Command's  Test  and  Evaluation  Functions", 
20  December  1972 

INSURV  Instruction  13100. 1A,  "Aircraft  Acceptance  Trials; 
Procedures  For",  1  October  1972 
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APPENDIX  D 
NAVAL  AIRCRAFT  WEAPON  SYSTEM  T&E  ORGANIZATIONS 


Naval  Air  Systems  Command 

Naval  Aerospace  Recovery  Facility,  El  Centro,  Calif. 
Develop,  test,  and  evaluate  parachutes  and  related 
systems  for  human  escape  methods  and  systems,  and 
retardation  and  recovery  systems. 

Naval  Air  Engineering  Center,  Philadelphia,  Pa. 

RDT&E  in  launching,  recovery,  and  landing  aids  for 
aircraft  and  in  ground  support  equipment  for  aircraft 
and  airborne  weapons  systems. 

Naval  Air  Propulsion  Test  Center,  Trenton,  N.J. 

Test  and  evaluate  aircraft  propulsion  systems  —  their 
components,  accessories,  fuels,  and  lubricants. 

Naval  Air  Test  Center,  Patuxent  River,  Md. 

Test  and  evaluation  of  aircraft  weapons  systems, 
their  components  and  related  equipment. 

Naval  Missile  Center,  Pt .  Mugu,  Calif. 

Test,  evaluation  and  development  support  of  Naval 
weapons,  weapons  systems  and  related  devices. 

Naval  Weapons  Evaluation  Facility,  Alburquerque ,  N.M. 

Test  and  evaluation  and  technical  support  for  nuclear 
and  designated  non  nuclear  weapons  and  weapon  systems. 

Operational  Test  and  Evaluation  Force 

VX-1  (Air  Test  and  Evaluation  Squadron  One),  Patuxent  River,  Md. 
OT&E  of  airborne  anti-submarine  weapon  systems,  support 
systems,  components,  and  equipment. 

VX-4,  Point  Mugu,  Calif. 

OT&E  of  all-weather  fighter  weapons  systems  and  air- 
launched  guided  missiles  weapon  systems. 

VX-5,  China  Lake,  Calif. 

OT&E  of  airborne  attack  weapon  systems  and  support  systems. 
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Chief  of  Naval  Research 

Naval  Air  Development  Center,  Warminster,  Pa. 

Principal  Navy  Research,  Development,  T&E  center  for 
Naval  Aircraft  systems. 

Naval  Weapons  Center,  China  Lake,  California 

Principal  Navy  RDT&E  center  for  air  warfare  and 
missile  systems. 
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APPENDIX  E 

NAVY  RANGES  AND  FACILITIES  USED 
FOR  AIRCRAFT  WEAPON  SYSTEM  T&E 

Ranges : 

Atlantic  Fleet  Weapons  Range,  Roosevelt  Roads,  Puerto 
Rico  " 

Atlantic  Undersea  T&E  Center  (AUTEC),  West  Palm  Beach, 
Fla. 

Pacific  Missile  Range  (PMR),  Pt .  Mugu,  Ca. 

Naval  Weapons  Center  (NWC)  Ranges,  China  Lake,  Ca. 

Naval  Weapons  Laboratory  (NWL)  Ranges,  Dahlgren,  Va . 

s 

Facilities  with  their  associated  test  cites: 

Naval  Aerospace  Recovery  Facility,  El  Centro,  Ca. 

Naval  Air  Propulsion  Test  Center,  Trenton,  N.J. 

Naval  Air  Test  Center  (NATC),  Patuxent  River,  Md. 

Naval  Air  Test  Facility  (NATF),  Lakehurst,  N.J. 

Naval  Explosive  Ordnance  Disposal  Facility,  Indian 
Head,  Md. 

Naval  Ordnance  Missile  Test  Facility,  White  Sands, 
N.M. 

Naval  Weapons  Evaluation  Facility  (NWEF),  Albuquerque, 
N.M. 
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APPENDIX  F 
SYSTEM  TEST  AND  EVALUATION  TRAINING  SCHOOLS  (STETS) 

The  main  purpose  of  the  STETS  would  be  to  teach  the 
basic  fundamentals  of  T&E  to  project  officers  and  civilian 
engineers/analysts  who  are  engaged  in  aircraft  weapon  system 
T&E.   Specifically,  each  major  test  center  would  have  a 
STETS  which  would  provide  courses  of  study  for  all  project 
officers  and  civilian  engineers/analysts  who  are  being 
assigned  to  T&E  jobs.   Each  test  center's  STETS  would  con- 
centrate on  teaching  the  fundamentals  of  the  particular 
warfare  specialties  of  that  test  center  (e.g.  STETS  China 
Lake  would  concentrate  in  the  areas  of  aircraft  attack 
systems,  air  to  ground  weapons,  electronic  countermeasures, 
and  reconnaissance). 

In  addition  to  teaching  the  technical  and  statistical 
aspects  of  T&E,  STETS  would  also  conduct  classes  dealing 
with  the  following:   the  test  center's  range/s  utilization 
procedures  and  capabilities,  the  Navy's  organization  for 
weapon  system  acquisition  and  the  role  that  the  test  center 
fulfills,  the  test  center's  funding  system  and  the  proce- 
dures that  must  be  followed  to  enable  it  to  function, 
technical  services  (instrumentation,  photo  coverage, 
computer  time,  etc.)  that  are  available  to  the  test  team 
and  hov;  to  obtain  those  services,  and  technical  report 
writing. 
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The  STETS  full  course  of  study  would  take  approximately 
eight  to  ten  weeks  with  half  day  attendance.   The  half  day 
attendance  would  enable  the  pilots  and  NFO's  to  spend  the 
other  halves  of  these  days  in  familiarizing  themselves  with 
the  aircraft/new  systems  they  would  eventually  be  testing, 
in  further  developing  their  flying  proficiency,  and  in 
commencing  an  orderly  project  takeover  from  the  project 
officer  they  would  be  replacing.   The  civilian  engineers/ 
analysts  would  spend  the  other  halves  of  these  days  working 
in  their  respective  divisions  by  providing  assistance  to 
the  experienced  engineers  in  data  reduction  or  in  some  other 
tasks  which  they  could  fulfill. 

Besides  the  STETS  full  course  of  study,  abbreviated 
courses  should  be  offered.   These  abbreviated  curriculums 
would  serve  as  refresher  courses  for  officers  who  are 
reassigned  to  T&E  billets  and  have  already  graduated  from 
the  full  STETS  course.   The  abbreviated  curriculum  could 
also  be  used  to  give  TPS  graduates  that  are  being  assigned 
to  non-aircraft  division  billets  a  better  foundation  in 
Weapon  System  testing. 

The  following  is  a  typical  course  content  which  could 
be  utilized  by  the  STETS. 

1.   Mathematics/Mechanics  refresher  -  introductory 
course  including  analytical  geometry,  calculus,  statics, 
and  dynamics.   Its  main  purpose  would  be  to  ensure  all 
students  have  a  reasonable  foundation  in  these  essential 
engineering  fundamentals. 
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2.  Statistics  -  a  course  to  give  the  student  sufficient 
depth  of  understanding  in  probability  and  statistics  that 

he  would  be  able  to  appreciate  the  statistical  implications 
of  test  design  and  test  results.   Course  would  present  some 
simple  statistical  test  designs  and  illustrate  the  increased 
test  effectiveness  and  potential  cost  savings  that  are 
possible  when  tests  are  designed  knowledgeably . 

3.  Computer  Technology  -  an  introductory  course  where 
digital  and  analog  systems  are  studied.   Fundamentals  would 
be  studied  and  student  would  be  taught  how  to  utilize 
various  computer  systems  available  at  the  test  center.   The 
integration  of  computers  into  complex  weapon  systems  would 
be  studied  as  would  methods  of  differentiating  computer 
software  problems  from  system  hardware  problems  in  complex 
weapon  system  testing. 

4.  Fundamentals  in  Warfare  Specialty  (more  than  one  of 
these  might  be  offered  at  each  STETS)  -  course  would  teach 
the  basic  fundamentals  of  the  particular  warfare  specialty. 
General  areas  to  be  studied  include:   test  techniques,  sys- 
tem integration  problems,  state  of  the  art  limitations, 

new  technological  advances  and  their  potential  applications, 
etc . 

5.  Data  acquisiton  -  course  would  study  different  types 
of  instrumentation  and  photo  equipment  systems.   Capabili- 
ties, limitations,  and  relative  expenses  of  each  would  be 
studied.   Each  student  would  learn  how  to  read  different 
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types  of  instrumentation  and  become  familiar  with  the 
typical  errors  that  may  result. 

6.  Weapon  System  Acquisition  Process  -  this  course 
would  present  a  general  overview  of  the  entire  acquisition 
process  highlighting  how  the  particular  test  center  fits 
into  the  process.   Requirements  determination,  DSARC  pro- 
cess, DOD  PPBS  process,  DOD  agency  roles  and  responsibili- 
ties, DOD  budget  cycle,  program  manager's  role,  etc.  would 
be  studied. 

7.  Range  Operation  -  operation  and  capabilities  of  the 
test  center's  ranges  would  be  studied  as  would  the  capabil- 
ities available  at  other  Navy  and  DOD  ranges.  The  particu- 
lar operating  procedures  for  the  test  center's  ranges  would 
be  studied. 

8.  Test  Center  Support  -  the  myriad  of  support  facili- 
ties at  the  test  center  would  be  studied  as  would  the  pro- 
cedures required  to  obtain  these  services.   These  include 
instrumentation,  photo  coverage,  computer  services,  aircraft 
maintenance,  etc. 

9.  Project  Funding  -  the  operation  of  the  test  center's 
budget  system  would  be  studied  as  would  the  procedures 
required  to  obtain  funding  for  a  particular  project.   The 
test  center's  budget  system  reporting  requirements  would 

be  taught. 

10.  Report  Writing  -  this  course  would  teach  technical 
report  writing  in  accordance  with  the  test  center's 
procedures . 
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